Oct 31 00:16:14.635116 ip-10-0-129-199 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 00:16:14.635130 ip-10-0-129-199 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 00:16:14.635140 ip-10-0-129-199 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 00:16:14.635460 ip-10-0-129-199 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 00:16:26.030859 ip-10-0-129-199 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 00:16:26.030878 ip-10-0-129-199 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e1c92a72893f4d30996a0100f1d422d0 -- Oct 31 00:18:31.391828 ip-10-0-129-199 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:18:31.825558 ip-10-0-129-199 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:18:31.825558 ip-10-0-129-199 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:18:31.825558 ip-10-0-129-199 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:18:31.825558 ip-10-0-129-199 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:18:31.829908 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829666 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:31.829908 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829902 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:31.829908 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829908 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:31.829908 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829912 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:31.829908 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829916 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:31.829908 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829920 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:31.829908 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829924 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829928 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829932 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829936 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829939 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829943 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829947 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829951 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829955 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829958 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829962 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829966 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829975 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829980 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829983 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829989 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829993 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.829997 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830001 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830005 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:31.830341 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830009 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830013 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830016 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830020 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830024 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830027 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830031 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830035 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830039 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830042 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830046 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830050 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830054 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830065 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830071 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830075 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830080 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830084 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830090 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:31.831166 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830094 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830100 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830104 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830108 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830113 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830117 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830121 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830126 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830130 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830134 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830139 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830143 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830147 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830151 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830160 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830168 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830173 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830178 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830183 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:31.832016 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830187 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830191 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830196 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830201 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830205 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830209 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830226 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830231 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830244 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830248 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830252 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830257 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830261 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830265 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830269 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830274 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830278 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830282 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830286 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:31.832820 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830290 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830297 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830302 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830307 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830312 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830316 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830321 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830325 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830329 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830334 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830337 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830341 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830345 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830349 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830353 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830360 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830364 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830370 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830374 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:31.833636 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830378 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830384 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830389 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.830396 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831859 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831872 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831878 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831885 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831890 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831894 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831900 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831905 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831909 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831914 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831918 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831923 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831927 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831932 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831937 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:31.834270 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831941 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831946 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831950 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831954 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831959 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831963 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831967 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831971 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831977 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831983 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831988 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831993 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.831997 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832001 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832005 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832009 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832014 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832019 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832023 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832027 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:31.834759 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832031 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832035 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832039 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832043 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832048 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832052 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832056 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832061 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832065 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832069 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832075 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832079 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832084 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832088 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832092 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832097 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832101 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832105 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832110 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:31.835419 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832114 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832118 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832123 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832127 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832131 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832135 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832140 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832144 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832149 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832153 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832157 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832164 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832168 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832172 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832177 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832181 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832185 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832189 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832195 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832202 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:31.835952 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832209 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832214 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832219 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832224 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832229 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832234 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832238 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832243 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832247 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832251 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832255 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832259 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832264 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832268 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832273 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832280 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832285 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832291 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832297 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:31.836462 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832302 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832309 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832314 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832319 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832323 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832328 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832333 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832337 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832342 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832346 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832350 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832354 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.832358 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833352 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833362 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833368 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833388 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833393 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833397 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:31.836963 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833402 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833407 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833412 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833416 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833420 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833424 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833428 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833433 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833438 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833442 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833446 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833468 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833473 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833477 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833482 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833486 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833491 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833495 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833500 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:31.837444 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833505 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833509 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833513 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833519 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833524 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833529 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833533 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833537 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833541 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833545 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833551 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833555 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833560 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833564 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833568 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833573 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833578 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833582 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833586 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833590 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:31.837945 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833594 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833599 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833603 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833609 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833615 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833621 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833626 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833630 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833634 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833638 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833642 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833646 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833651 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833655 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833659 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833663 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833667 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833672 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833676 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:31.838417 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833680 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833685 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833689 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833694 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833698 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833702 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833706 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833710 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833714 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833719 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833723 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833729 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833733 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833737 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833742 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833746 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833750 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833754 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833758 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833762 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:31.838926 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833767 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833772 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833776 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833780 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833785 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833789 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833794 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833798 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833802 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833807 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833812 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833816 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833821 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833826 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833830 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833834 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833839 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833843 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833847 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833852 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:31.839400 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833856 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.833860 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835015 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835033 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835043 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835052 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835060 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835065 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835073 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835080 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835085 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835091 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835097 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835102 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835107 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835113 2572 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835118 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835123 2572 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835128 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835135 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835143 2572 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835148 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835153 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835158 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:18:31.840040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835164 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835170 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835176 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835184 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835190 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835195 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835200 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835205 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835210 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835215 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835222 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835228 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835232 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835238 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835242 2572 flags.go:64] FLAG: --enable-server="true" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835247 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835254 2572 flags.go:64] FLAG: --event-burst="100" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835260 2572 flags.go:64] FLAG: --event-qps="50" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835265 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835270 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835275 2572 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835282 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835287 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835292 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835297 2572 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:18:31.840672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835302 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835307 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835312 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835318 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835323 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835328 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835333 2572 flags.go:64] FLAG: --feature-gates="" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835340 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835345 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835350 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835358 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835363 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835368 2572 flags.go:64] FLAG: --help="false" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835373 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-129-199.ec2.internal" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835379 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835385 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835390 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835434 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835444 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835448 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835467 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835473 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835479 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:18:31.841295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835485 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835489 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835492 2572 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835497 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835500 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835504 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835507 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835510 2572 flags.go:64] FLAG: --lock-file="" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835513 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835517 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835522 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835530 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835533 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835537 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835540 2572 flags.go:64] FLAG: --logging-format="text" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835543 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835547 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835550 2572 flags.go:64] FLAG: --manifest-url="" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835553 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835560 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835564 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835568 2572 flags.go:64] FLAG: --max-pods="110" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835572 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835575 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835579 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:18:31.841900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835582 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835586 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835590 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835594 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835601 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835604 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835608 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835612 2572 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835615 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835618 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835621 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835624 2572 flags.go:64] FLAG: --port="10250" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835628 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835631 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03c16c5961cd995d9" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835634 2572 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835637 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835640 2572 flags.go:64] FLAG: --register-node="true" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835644 2572 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835648 2572 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835652 2572 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835655 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835658 2572 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835662 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835665 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835669 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835672 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:18:31.842915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835675 2572 flags.go:64] FLAG: --runonce="false" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835679 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835682 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835685 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835688 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835692 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835698 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835702 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835706 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835710 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835713 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835716 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835720 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835723 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835726 2572 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835729 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835735 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835738 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835741 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835748 2572 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835750 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835753 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835757 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835760 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835764 2572 flags.go:64] FLAG: --v="2" Oct 31 00:18:31.843962 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835769 2572 flags.go:64] FLAG: --version="false" Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835774 2572 flags.go:64] FLAG: --vmodule="" Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835779 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.835782 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835934 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835939 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835942 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835945 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835948 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835951 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835954 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835957 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835960 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835966 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835969 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835972 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835975 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835978 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835981 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835984 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:31.845216 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835987 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835990 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835992 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835995 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.835997 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836000 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836003 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836005 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836008 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836011 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836013 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836016 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836020 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836024 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836028 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836031 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836034 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836037 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836041 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:31.846078 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836045 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836048 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836050 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836053 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836056 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836058 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836063 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836066 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836068 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836071 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836074 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836077 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836080 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836082 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836085 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836088 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836091 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836094 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836097 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836099 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:31.846916 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836102 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836105 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836108 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836111 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836113 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836116 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836119 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836121 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836124 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836127 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836130 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836132 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836135 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836137 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836140 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836143 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836145 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836148 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836156 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:31.847818 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836159 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836161 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836164 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836167 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836170 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836174 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836177 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836179 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836182 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836185 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836187 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836190 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836193 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836196 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836199 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836202 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836205 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836207 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836210 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:31.848375 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836212 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836215 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836218 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836221 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836223 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836226 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836229 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836232 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836235 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836237 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836240 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836243 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836248 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.836278 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836320 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836326 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:31.849118 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836330 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836333 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836336 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836340 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836343 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836345 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836348 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836351 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836354 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836357 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836360 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836363 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836366 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836368 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836371 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836374 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836376 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836379 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836382 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836385 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:31.849866 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836387 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836390 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836392 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836395 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836398 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836401 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836403 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836406 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836411 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836415 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836418 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836429 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836432 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836434 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836437 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836441 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836444 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836447 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836464 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836467 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:31.850395 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836471 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836473 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836476 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836479 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836483 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836485 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836490 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836493 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836495 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836498 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836501 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836504 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836507 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836509 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836512 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836514 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836517 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836520 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836522 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836525 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:31.850902 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836529 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836533 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836535 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836538 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836540 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836543 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836546 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836548 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836552 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836557 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836560 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836564 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836566 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836569 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836573 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836577 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836580 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836582 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836585 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:31.851535 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836588 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836590 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836593 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836595 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836598 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836600 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836603 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836605 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836608 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836611 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836613 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836616 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836618 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836623 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836626 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836628 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836631 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836634 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836637 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:31.852125 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836640 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836642 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836645 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836648 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836651 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.836653 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.845779 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.845808 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.845832 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.845841 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845940 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845948 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845953 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845957 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845961 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845966 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845970 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845975 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845979 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:31.852641 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845983 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845988 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845992 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.845996 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846000 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846005 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846009 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846013 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846018 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846022 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846026 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846030 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846034 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846039 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846043 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846047 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846053 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846057 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846063 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846067 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:31.853140 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846073 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846078 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846082 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846086 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846090 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846094 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846098 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846103 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846107 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846111 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846116 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846120 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846124 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846129 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846133 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846137 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846142 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846147 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846151 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846156 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:31.853690 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846160 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846165 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846169 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846174 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846179 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846183 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846188 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846192 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846196 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846201 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846205 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846210 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846215 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846219 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846223 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846228 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846232 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846237 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846241 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846245 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:31.854160 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846250 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846254 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846258 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846263 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846267 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846271 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846275 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846279 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846284 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846289 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846294 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846298 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846303 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846307 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846311 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846315 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846320 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846324 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846328 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:31.854802 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846333 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846337 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846342 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846348 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846356 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846363 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846369 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846375 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846380 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846384 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846390 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846395 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846399 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846404 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846409 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846413 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846417 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:31.855263 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846421 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.846490 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846736 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846746 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846752 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846760 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846767 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846774 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846779 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846783 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846788 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846792 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846797 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846801 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846806 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:31.855788 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846810 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846814 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846819 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846823 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846827 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846831 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846836 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846840 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846845 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846849 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846854 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846858 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846862 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846866 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846870 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846875 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846879 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846883 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846888 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:31.856181 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846892 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846897 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846902 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846906 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846910 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846915 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846921 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846925 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846930 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846933 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846937 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846942 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846946 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846950 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846954 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846959 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846963 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846967 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846972 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:31.856703 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846976 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846980 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846984 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846988 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846993 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.846997 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847001 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847006 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847010 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847015 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847019 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847023 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847027 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847031 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847035 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847040 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847045 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847049 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847053 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847058 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:31.857154 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847063 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847068 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847072 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847076 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847081 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847085 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847089 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847094 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847099 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847103 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847107 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847111 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847115 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847120 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847124 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847128 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847133 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847137 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847141 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:31.857653 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847145 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847149 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847153 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847157 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847161 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847166 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847170 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847175 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847182 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847188 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847193 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847197 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847201 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847209 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847214 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847218 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:31.858104 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.847262 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847319 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847327 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847332 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847337 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847342 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847346 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847351 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847355 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847359 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847363 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847368 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847372 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847376 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847380 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847385 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847390 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847394 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847400 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847406 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:18:31.858533 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847410 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847414 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847421 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847425 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847430 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847434 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847439 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847443 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847447 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847470 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847478 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847484 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847489 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847494 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847498 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847503 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847508 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847512 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847516 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847521 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:18:31.859021 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847525 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847529 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847534 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847538 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847542 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847546 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847551 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847555 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847561 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847565 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847570 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847574 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847578 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847583 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847589 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847594 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847598 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847602 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847607 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:18:31.859575 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847611 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847615 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847620 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847626 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847631 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847637 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847641 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847646 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847650 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847654 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847659 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847663 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847667 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847672 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847676 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847680 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847685 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847689 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847693 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:18:31.860053 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847697 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847702 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847706 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847710 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847714 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847719 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847723 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847728 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847733 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847737 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847742 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847746 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847751 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847755 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847759 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847763 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847769 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847774 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847779 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847784 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:18:31.860536 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847788 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847793 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847797 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847802 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847806 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847810 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847817 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847821 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:31.847825 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.847917 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.851918 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.853181 2572 server.go:1014] "Starting client certificate rotation" Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.853301 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.853349 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.855926 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:18:31.861013 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.858814 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:18:31.870779 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.870752 2572 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:18:31.873761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.873743 2572 log.go:25] "Validated CRI v1 image API" Oct 31 00:18:31.874742 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.874717 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:18:31.880123 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.880096 2572 fs.go:135] Filesystem UUIDs: map[470fd304-ba8e-4cc8-b4d7-d6565a532618:/dev/nvme0n1p4 62f55614-cd56-434a-a03c-273f37a7fe4d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 00:18:31.880123 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.880119 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:18:31.886599 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.886437 2572 manager.go:217] Machine: {Timestamp:2026-10-31 00:18:31.8846794 +0000 UTC m=+0.364855974 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098612 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cdd1a4884d5506c6853617cbaed5a SystemUUID:ec2cdd1a-4884-d550-6c68-53617cbaed5a BootID:e1c92a72-893f-4d30-996a-0100f1d422d0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cc:6f:6a:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cc:6f:6a:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:e0:d1:0e:16:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:18:31.886599 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.886591 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:18:31.886721 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.886693 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:18:31.889641 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.889603 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:18:31.889799 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.889644 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-199.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:18:31.889849 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.889802 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:18:31.889849 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.889810 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:18:31.889904 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.889849 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:18:31.889904 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.889871 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:18:31.890764 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.890751 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:18:31.894536 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.894519 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:18:31.894674 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.894665 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:18:31.895501 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.895491 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:18:31.895546 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.895505 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:18:31.895546 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.895526 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:18:31.895546 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.895535 2572 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:18:31.895546 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.895546 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:18:31.896008 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.895991 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:18:31.898514 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.898480 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:18:31.904074 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.904055 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:18:31.904831 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.904817 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:18:31.904874 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.904853 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:18:31.905044 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905031 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:18:31.905081 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905055 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:18:31.905081 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905070 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:18:31.905136 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905086 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:18:31.905136 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905097 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:18:31.905136 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905105 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:18:31.905570 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905560 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:18:31.905604 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905571 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:18:31.905604 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905578 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:18:31.905604 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905584 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:18:31.905604 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905594 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:18:31.905604 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.905602 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:18:31.907169 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.907158 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:18:31.907169 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.907169 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:18:31.911188 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.911163 2572 server.go:1263] "Started kubelet" Oct 31 00:18:31.911408 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.911342 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:18:31.911981 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.911319 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:18:31.912077 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.912066 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:18:31.912389 ip-10-0-129-199 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:18:31.913658 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.913629 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:18:31.918329 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.917831 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:18:31.920710 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.920657 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:18:31.921805 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.921776 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:18:31.922390 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.922370 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:18:31.922735 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.922715 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found" Oct 31 00:18:31.923101 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923083 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:18:31.923101 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923102 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:18:31.923245 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923230 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:18:31.923245 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923242 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:18:31.923342 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923255 2572 factory.go:55] Registering systemd factory Oct 31 00:18:31.923342 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923271 2572 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:18:31.923499 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923480 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:18:31.923573 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923514 2572 factory.go:153] Registering CRI-O factory Oct 31 00:18:31.923573 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923531 2572 factory.go:223] Registration of the crio container factory successfully Oct 31 00:18:31.923667 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923583 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:18:31.923667 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923613 2572 factory.go:103] Registering Raw factory Oct 31 00:18:31.923667 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.923627 2572 manager.go:1199] Started watching for new ooms in manager Oct 31 00:18:31.924646 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.924627 2572 manager.go:319] Starting recovery of all containers Oct 31 00:18:31.935441 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.935415 2572 manager.go:324] Recovery completed Oct 31 00:18:31.941326 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.941286 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:31.943515 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.943493 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:31.943630 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.943525 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:31.943630 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.943536 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:31.944096 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.944078 2572 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:18:31.944096 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.944093 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:18:31.944223 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.944112 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:18:31.944223 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.944084 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:31.944991 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.944972 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:31.945073 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.945000 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:31.945073 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.945010 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:31.946390 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.946375 2572 policy_none.go:50] "Start" Oct 31 00:18:31.946390 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.946392 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:18:31.946520 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.946403 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:18:31.948557 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.948542 2572 policy_none.go:44] "Start" Oct 31 00:18:31.951932 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.950605 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-199.ec2.internal.18e3752ccd8c4ba3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-199.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-199.ec2.internal,},FirstTimestamp:2026-10-31 00:18:31.911123875 +0000 UTC m=+0.391300439,LastTimestamp:2026-10-31 00:18:31.911123875 +0000 UTC m=+0.391300439,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-199.ec2.internal,}" Oct 31 00:18:31.954058 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.954037 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:18:31.954125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.954088 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bt4p5" Oct 31 00:18:31.962259 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.962224 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-199.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:18:31.962259 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.962223 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:18:31.965667 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.965633 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bt4p5" Oct 31 00:18:31.973520 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.973433 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-199.ec2.internal.18e3752ccf7a801b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-199.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-199.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-199.ec2.internal,},FirstTimestamp:2026-10-31 00:18:31.943512091 +0000 UTC m=+0.423688655,LastTimestamp:2026-10-31 00:18:31.943512091 +0000 UTC m=+0.423688655,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-199.ec2.internal,}" Oct 31 00:18:31.973637 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.973500 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 00:18:31.973948 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.973927 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 00:18:31.985246 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.985192 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:18:31.988210 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.988193 2572 manager.go:341] "Starting Device Plugin manager" Oct 31 00:18:31.988349 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.988229 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:18:31.988349 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.988239 2572 server.go:87] "Starting device plugin registration server" Oct 31 00:18:31.988566 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.988552 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:18:31.988660 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.988565 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:18:31.988715 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.988696 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:18:31.988859 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.988781 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:18:31.988859 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:31.988794 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:18:31.989706 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.989434 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:18:31.989706 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:31.989517 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-199.ec2.internal\" not found" Oct 31 00:18:32.001580 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.001544 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:32.055565 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.055528 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:18:32.057259 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.056809 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:18:32.057259 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.056838 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:18:32.057259 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.056860 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:18:32.057259 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:32.056893 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:18:32.071526 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.071490 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:32.089444 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.089342 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:18:32.092137 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.092114 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:18:32.092268 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.092151 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:18:32.092268 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.092162 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:18:32.092268 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.092195 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.109818 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.109788 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.157583 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.157511 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"] Oct 31 00:18:32.159883 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.159863 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.161547 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.161530 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.192832 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.192813 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.195767 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.195751 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.225025 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.224992 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.225025 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.225026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.225203 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.225044 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad1af2674e0c55384c949adc8f5b6f9c-config\") pod \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" (UID: \"ad1af2674e0c55384c949adc8f5b6f9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.227391 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.227376 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:18:32.230323 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.230307 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:18:32.326536 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.326488 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.326691 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.326555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.326691 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.326590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad1af2674e0c55384c949adc8f5b6f9c-config\") pod \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" (UID: \"ad1af2674e0c55384c949adc8f5b6f9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.326691 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.326622 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.326691 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.326645 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad1af2674e0c55384c949adc8f5b6f9c-config\") pod \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" (UID: \"ad1af2674e0c55384c949adc8f5b6f9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.326691 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.326658 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.532368 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.532260 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.534632 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.534608 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:18:32.853565 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.853448 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:18:32.854115 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.853685 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:18:32.895994 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.895947 2572 apiserver.go:52] "Watching apiserver" Oct 31 00:18:32.922558 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.922514 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:18:32.949715 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.949683 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:18:32.950081 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.950057 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-xzjcg","openshift-network-operator/iptables-alerter-b2slc","kube-system/global-pull-secret-syncer-z4hfm","openshift-dns/node-resolver-npncc","kube-system/konnectivity-agent-bshvl","kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal","openshift-network-diagnostics/network-check-target-mswcp","openshift-ovn-kubernetes/ovnkube-node-sgkkv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f","openshift-cluster-node-tuning-operator/tuned-gffpv","openshift-image-registry/node-ca-jpxt2","openshift-multus/multus-additional-cni-plugins-mhppg","openshift-multus/multus-bfsdv"] Oct 31 00:18:32.952664 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.952634 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:32.952800 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:32.952739 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:32.954976 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.954950 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:32.957184 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.957159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:32.957294 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:32.957237 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:32.959270 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.959247 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:32.961293 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.961274 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:32.963445 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.963422 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:32.963552 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:32.963502 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:32.965744 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.965726 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:32.968052 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.968031 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:32.970973 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.970348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:32.972197 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.972123 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:13:32 +0000 UTC" deadline="2028-05-31 05:00:44.055112616 +0000 UTC" Oct 31 00:18:32.972252 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.972198 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13876h42m11.082918021s" Oct 31 00:18:32.972780 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.972769 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:32.975335 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.975313 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:32.975651 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.975474 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-grldd\"" Oct 31 00:18:32.976364 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.976345 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:32.976438 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.976380 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:18:32.976438 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.976398 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:18:32.977686 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.977664 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bfsdv" Oct 31 00:18:32.977780 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.977743 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:18:32.978345 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:32.978326 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:18:33.005635 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.005604 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgmpl\"" Oct 31 00:18:33.015877 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.015853 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:33.023680 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.023645 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:18:33.024411 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.024168 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:18:33.024411 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.024190 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:18:33.024411 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.024304 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:18:33.024595 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.024573 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:18:33.024944 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.024917 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:18:33.026851 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.026833 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:18:33.026948 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.026927 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:18:33.027925 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.027911 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:18:33.028029 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.028009 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:18:33.028091 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.028022 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:18:33.028271 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.028249 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:18:33.028381 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.028297 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wlc7\"" Oct 31 00:18:33.028476 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.028450 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:18:33.028545 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.028505 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rqjpt\"" Oct 31 00:18:33.028545 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.028530 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:18:33.028667 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.028650 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jw4zk\"" Oct 31 00:18:33.028703 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.028676 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4w2kf\"" Oct 31 00:18:33.030379 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.030362 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dqh4t\"" Oct 31 00:18:33.032703 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.032684 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flpkx\" (UniqueName: \"kubernetes.io/projected/47350ce3-ba5e-4393-ac0f-653b41bc28fd-kube-api-access-flpkx\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.032780 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.032716 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfq6z\" (UniqueName: \"kubernetes.io/projected/2b1af480-ae8f-483e-bd52-947094fcc3ff-kube-api-access-jfq6z\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.032836 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.032783 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysconfig\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.033256 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033234 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99f8918-3e3d-456b-be91-db4120b67424-host\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.033351 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033277 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-sys\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.033405 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-script-lib\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.033477 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-socket-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.033477 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033441 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-registration-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.033588 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033482 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-modprobe-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.033588 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033521 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-env-overrides\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.033588 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033564 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rbb\" (UniqueName: \"kubernetes.io/projected/064705a9-54fd-4528-ae88-1cb4596c5b6e-kube-api-access-j8rbb\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.033820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033782 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-kubelet-config\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:33.033879 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033846 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-kubelet\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.033935 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033898 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-var-lib-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.033986 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033931 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-kubelet\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.034042 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.033992 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-hostroot\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.034095 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-log-socket\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.034095 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034084 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-system-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.034193 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034118 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.034268 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034254 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.034322 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034276 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-tmp\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.034322 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034298 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-systemd-units\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.034416 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034320 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-bin\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.034416 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034388 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-multus-certs\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.034416 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034410 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.034574 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034428 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.034574 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034449 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-etc-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.034574 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034494 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-node-log\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.034773 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.034773 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034649 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-bin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.034773 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034681 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-conf-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.034921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034841 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f082d-9176-491a-8d5e-281ee6665dae-konnectivity-ca\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:33.034921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-cnibin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.035019 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034919 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-k8s-cni-cncf-io\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.035019 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47350ce3-ba5e-4393-ac0f-653b41bc28fd-hosts-file\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.035019 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.034996 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-run\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.035175 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035027 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:33.035175 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035054 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47350ce3-ba5e-4393-ac0f-653b41bc28fd-tmp-dir\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.035175 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035106 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-device-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.035175 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035141 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-lib-modules\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.035175 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035171 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-tuned\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.035419 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035224 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-cnibin\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.035419 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035276 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.035419 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035298 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-slash\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.035419 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035316 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-ovn\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.035419 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-netd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.035674 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035637 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qh9\" (UniqueName: \"kubernetes.io/projected/f99f8918-3e3d-456b-be91-db4120b67424-kube-api-access-76qh9\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.035729 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035677 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-kubernetes\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.035729 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035703 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-var-lib-kubelet\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.035832 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfzq2\" (UniqueName: \"kubernetes.io/projected/05e75b3b-ffb1-4cda-9203-f4882bd385e7-kube-api-access-tfzq2\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.035832 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035775 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x95q\" (UniqueName: \"kubernetes.io/projected/57df0df3-a039-45eb-8ae2-02433f6b0b2e-kube-api-access-8x95q\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:33.035832 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.035977 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035845 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.035977 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-config\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.035977 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-netns\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.035977 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99f8918-3e3d-456b-be91-db4120b67424-serviceca\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.036167 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.035976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/3af6b8f1-b029-4ae2-99cb-df0315daa268-kube-api-access-5dlwf\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.036167 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.036008 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-sys-fs\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.036167 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.036040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-host\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.036167 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.036075 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-system-cni-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.036167 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.036107 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5nl\" (UniqueName: \"kubernetes.io/projected/2d60169a-68c9-4cfd-bbd7-73ecac659695-kube-api-access-2v5nl\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.036505 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.036483 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-cni-binary-copy\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.036583 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.036553 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-netns\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.036684 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.036632 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-multus\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.036908 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.036866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-daemon-config\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.036993 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.036963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-etc-kubernetes\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.037095 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037081 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b70404c4-e809-42c9-9efc-d6262e52815c-host-slash\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.037156 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037140 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-conf\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.037205 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.037566 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037250 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.037639 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037600 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-systemd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.037709 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037649 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f9f082d-9176-491a-8d5e-281ee6665dae-agent-certs\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:33.037709 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037677 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-systemd\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.037785 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:33.037785 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-dbus\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:33.037855 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037782 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:33.037855 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037820 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.037952 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037857 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b70404c4-e809-42c9-9efc-d6262e52815c-iptables-alerter-script\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.037952 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037902 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sk42\" (UniqueName: \"kubernetes.io/projected/b70404c4-e809-42c9-9efc-d6262e52815c-kube-api-access-2sk42\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.038046 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037960 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovn-node-metrics-cert\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.038046 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.037987 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-socket-dir-parent\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.050104 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.050080 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:33.057470 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.057444 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:18:33.057614 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.057524 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qmhtp" Oct 31 00:18:33.070822 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.070798 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qmhtp" Oct 31 00:18:33.084338 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084069 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9prwf\"" Oct 31 00:18:33.084461 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084123 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pp79q\"" Oct 31 00:18:33.084461 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084173 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:18:33.084461 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084373 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:18:33.084461 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084215 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:18:33.084461 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084239 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:18:33.084461 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084307 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:18:33.084461 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084373 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:18:33.084461 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084376 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:18:33.084461 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.084197 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:18:33.138529 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b70404c4-e809-42c9-9efc-d6262e52815c-host-slash\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.138529 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138492 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-conf\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.138529 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138535 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-systemd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138575 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f9f082d-9176-491a-8d5e-281ee6665dae-agent-certs\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138574 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b70404c4-e809-42c9-9efc-d6262e52815c-host-slash\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138630 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-systemd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138693 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-conf\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-systemd\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138809 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-dbus\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138834 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138847 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-systemd\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138860 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b70404c4-e809-42c9-9efc-d6262e52815c-iptables-alerter-script\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sk42\" (UniqueName: \"kubernetes.io/projected/b70404c4-e809-42c9-9efc-d6262e52815c-kube-api-access-2sk42\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.138921 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138927 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-dbus\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:33.139652 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.138937 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:33.139652 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.139015 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs podName:57df0df3-a039-45eb-8ae2-02433f6b0b2e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:33.638990083 +0000 UTC m=+2.119166652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs") pod "network-metrics-daemon-xzjcg" (UID: "57df0df3-a039-45eb-8ae2-02433f6b0b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:33.139652 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.138937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovn-node-metrics-cert\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.139652 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139562 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-socket-dir-parent\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.139652 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flpkx\" (UniqueName: \"kubernetes.io/projected/47350ce3-ba5e-4393-ac0f-653b41bc28fd-kube-api-access-flpkx\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.139652 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139630 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq6z\" (UniqueName: \"kubernetes.io/projected/2b1af480-ae8f-483e-bd52-947094fcc3ff-kube-api-access-jfq6z\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139628 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139678 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b70404c4-e809-42c9-9efc-d6262e52815c-iptables-alerter-script\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139683 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139728 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139749 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-socket-dir-parent\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139745 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139807 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysconfig\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99f8918-3e3d-456b-be91-db4120b67424-host\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-sys\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.139895 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139894 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-script-lib\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139918 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysconfig\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139929 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99f8918-3e3d-456b-be91-db4120b67424-host\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.139952 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-sys\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140027 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-socket-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-registration-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-modprobe-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140194 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-socket-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140212 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-registration-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140164 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-env-overrides\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rbb\" (UniqueName: \"kubernetes.io/projected/064705a9-54fd-4528-ae88-1cb4596c5b6e-kube-api-access-j8rbb\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140287 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-modprobe-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-kubelet-config\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140355 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-kubelet\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-var-lib-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.140407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-kubelet\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-hostroot\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140455 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-log-socket\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140480 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-system-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140509 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-var-lib-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140536 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140580 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-kubelet-config\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140576 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-kubelet\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-tmp\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140628 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-log-socket\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140653 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-systemd-units\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140679 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-env-overrides\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140689 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140698 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-bin\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140679 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-kubelet\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140712 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-system-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.141116 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140720 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-hostroot\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140585 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-script-lib\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-multus-certs\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140760 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-systemd-units\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140771 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140795 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-bin\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140763 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-multus-certs\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140855 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-etc-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140894 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-node-log\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140925 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140953 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-node-log\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140956 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-etc-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.140995 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141027 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-bin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141053 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-conf-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141055 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141078 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f082d-9176-491a-8d5e-281ee6665dae-konnectivity-ca\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141101 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-conf-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141108 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-bin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141104 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-cnibin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141153 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-k8s-cni-cncf-io\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141160 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-cnibin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47350ce3-ba5e-4393-ac0f-653b41bc28fd-hosts-file\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-k8s-cni-cncf-io\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-run\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141250 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141262 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47350ce3-ba5e-4393-ac0f-653b41bc28fd-hosts-file\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141274 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47350ce3-ba5e-4393-ac0f-653b41bc28fd-tmp-dir\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-device-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141339 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-lib-modules\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-tuned\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-cnibin\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141444 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.142761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-slash\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141513 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-ovn\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141557 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-netd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qh9\" (UniqueName: \"kubernetes.io/projected/f99f8918-3e3d-456b-be91-db4120b67424-kube-api-access-76qh9\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141602 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-run\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-kubernetes\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141665 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f082d-9176-491a-8d5e-281ee6665dae-konnectivity-ca\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141674 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-var-lib-kubelet\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.141685 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141608 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47350ce3-ba5e-4393-ac0f-653b41bc28fd-tmp-dir\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzq2\" (UniqueName: \"kubernetes.io/projected/05e75b3b-ffb1-4cda-9203-f4882bd385e7-kube-api-access-tfzq2\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.141752 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret podName:a56c984b-c9d9-4d3f-86fd-2314f01a6673 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:33.641728945 +0000 UTC m=+2.121905514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret") pod "global-pull-secret-syncer-z4hfm" (UID: "a56c984b-c9d9-4d3f-86fd-2314f01a6673") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141769 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-kubernetes\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141788 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x95q\" (UniqueName: \"kubernetes.io/projected/57df0df3-a039-45eb-8ae2-02433f6b0b2e-kube-api-access-8x95q\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141864 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-lib-modules\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.143388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-config\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141914 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-device-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-netns\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99f8918-3e3d-456b-be91-db4120b67424-serviceca\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141957 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141965 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-var-lib-kubelet\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.141985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/3af6b8f1-b029-4ae2-99cb-df0315daa268-kube-api-access-5dlwf\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142028 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-sys-fs\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-host\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142085 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-system-cni-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142112 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5nl\" (UniqueName: \"kubernetes.io/projected/2d60169a-68c9-4cfd-bbd7-73ecac659695-kube-api-access-2v5nl\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142139 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-cni-binary-copy\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142153 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142169 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-netns\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142211 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-netns\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142210 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-multus\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142255 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-slash\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142258 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-cnibin\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142320 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-multus\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142321 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-host\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142364 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-netns\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142412 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-sys-fs\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142424 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.142256 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-daemon-config\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.143126 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-netd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.143175 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-ovn\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.143664 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-daemon-config\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.143722 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-system-cni-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.143758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-etc-kubernetes\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.143829 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-etc-kubernetes\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.143992 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-config\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.144168 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovn-node-metrics-cert\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.144225 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f9f082d-9176-491a-8d5e-281ee6665dae-agent-certs\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.144272 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99f8918-3e3d-456b-be91-db4120b67424-serviceca\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.144400 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-cni-binary-copy\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.144747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.144449 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-tmp\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.146344 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.146318 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-tuned\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.163479 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.163437 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:33.163479 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.163468 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:33.163479 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.163485 2572 projected.go:196] Error preparing data for projected volume kube-api-access-chgxt for pod openshift-network-diagnostics/network-check-target-mswcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:33.163791 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.163587 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt podName:6249a4c3-cd70-4c06-aa16-9864e6493086 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:33.663569473 +0000 UTC m=+2.143746049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-chgxt" (UniqueName: "kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt") pod "network-check-target-mswcp" (UID: "6249a4c3-cd70-4c06-aa16-9864e6493086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:33.165964 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.165934 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sk42\" (UniqueName: \"kubernetes.io/projected/b70404c4-e809-42c9-9efc-d6262e52815c-kube-api-access-2sk42\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.172347 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.171593 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rbb\" (UniqueName: \"kubernetes.io/projected/064705a9-54fd-4528-ae88-1cb4596c5b6e-kube-api-access-j8rbb\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.172347 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.171631 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flpkx\" (UniqueName: \"kubernetes.io/projected/47350ce3-ba5e-4393-ac0f-653b41bc28fd-kube-api-access-flpkx\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.172347 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.172275 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfq6z\" (UniqueName: \"kubernetes.io/projected/2b1af480-ae8f-483e-bd52-947094fcc3ff-kube-api-access-jfq6z\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.174295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.174269 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x95q\" (UniqueName: \"kubernetes.io/projected/57df0df3-a039-45eb-8ae2-02433f6b0b2e-kube-api-access-8x95q\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:33.174429 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.174340 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/3af6b8f1-b029-4ae2-99cb-df0315daa268-kube-api-access-5dlwf\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.174429 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.174344 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qh9\" (UniqueName: \"kubernetes.io/projected/f99f8918-3e3d-456b-be91-db4120b67424-kube-api-access-76qh9\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.174429 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.174383 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzq2\" (UniqueName: \"kubernetes.io/projected/05e75b3b-ffb1-4cda-9203-f4882bd385e7-kube-api-access-tfzq2\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.176711 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.176683 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5nl\" (UniqueName: \"kubernetes.io/projected/2d60169a-68c9-4cfd-bbd7-73ecac659695-kube-api-access-2v5nl\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.273899 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:33.273858 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b869eb4ef8b2ca62aee88ee86a131f.slice/crio-feae4182e7d2d731c263c8b3ddfa1e81e707fef6583ba1cb18aa78ed7682c00f WatchSource:0}: Error finding container feae4182e7d2d731c263c8b3ddfa1e81e707fef6583ba1cb18aa78ed7682c00f: Status 404 returned error can't find the container with id feae4182e7d2d731c263c8b3ddfa1e81e707fef6583ba1cb18aa78ed7682c00f Oct 31 00:18:33.279284 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:33.279260 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1af2674e0c55384c949adc8f5b6f9c.slice/crio-f95f9e90756a204a140abb4e56732e931cda0f25c18a8d2ba3d504610a3c90df WatchSource:0}: Error finding container f95f9e90756a204a140abb4e56732e931cda0f25c18a8d2ba3d504610a3c90df: Status 404 returned error can't find the container with id f95f9e90756a204a140abb4e56732e931cda0f25c18a8d2ba3d504610a3c90df Oct 31 00:18:33.281321 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.281275 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:18:33.284276 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.284227 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:18:33.291569 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.291541 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-npncc" Oct 31 00:18:33.296247 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:33.296219 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70404c4_e809_42c9_9efc_d6262e52815c.slice/crio-6c40f3a146031134a5ef8f7edad95480466c4028e385fabc50b348ddf6742a54 WatchSource:0}: Error finding container 6c40f3a146031134a5ef8f7edad95480466c4028e385fabc50b348ddf6742a54: Status 404 returned error can't find the container with id 6c40f3a146031134a5ef8f7edad95480466c4028e385fabc50b348ddf6742a54 Oct 31 00:18:33.297205 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.297175 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:33.302672 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:33.302647 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47350ce3_ba5e_4393_ac0f_653b41bc28fd.slice/crio-32d47e0e1ce59bc6c4787f4b8e8028b6abf07054417ca8d6c0d4838b95d05581 WatchSource:0}: Error finding container 32d47e0e1ce59bc6c4787f4b8e8028b6abf07054417ca8d6c0d4838b95d05581: Status 404 returned error can't find the container with id 32d47e0e1ce59bc6c4787f4b8e8028b6abf07054417ca8d6c0d4838b95d05581 Oct 31 00:18:33.308034 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:33.308002 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9f082d_9176_491a_8d5e_281ee6665dae.slice/crio-5f002c7dbb510dd14f973c08630a802a79cc6ae52e621603bd4d9d78d16292eb WatchSource:0}: Error finding container 5f002c7dbb510dd14f973c08630a802a79cc6ae52e621603bd4d9d78d16292eb: Status 404 returned error can't find the container with id 5f002c7dbb510dd14f973c08630a802a79cc6ae52e621603bd4d9d78d16292eb Oct 31 00:18:33.342724 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.342693 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:33.347809 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.347781 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:18:33.350217 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.350191 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:18:33.355353 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:33.355324 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064705a9_54fd_4528_ae88_1cb4596c5b6e.slice/crio-eab2e57560f114f1accf98cdb298edb1a4f619f6f09723003a23cd02a81f0a33 WatchSource:0}: Error finding container eab2e57560f114f1accf98cdb298edb1a4f619f6f09723003a23cd02a81f0a33: Status 404 returned error can't find the container with id eab2e57560f114f1accf98cdb298edb1a4f619f6f09723003a23cd02a81f0a33 Oct 31 00:18:33.361848 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:33.361794 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e75b3b_ffb1_4cda_9203_f4882bd385e7.slice/crio-ff642c15ac0a65be45e23166e5ef3057d41b6735d8489551e522baf6bc44d32e WatchSource:0}: Error finding container ff642c15ac0a65be45e23166e5ef3057d41b6735d8489551e522baf6bc44d32e: Status 404 returned error can't find the container with id ff642c15ac0a65be45e23166e5ef3057d41b6735d8489551e522baf6bc44d32e Oct 31 00:18:33.368956 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.368922 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:18:33.395300 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.395208 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bfsdv" Oct 31 00:18:33.401582 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.401545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:18:33.407310 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:33.407283 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af6b8f1_b029_4ae2_99cb_df0315daa268.slice/crio-c24bf0edadced92dbed653eb1521f7d88b866b801b4dbfd222bd9314aaf51849 WatchSource:0}: Error finding container c24bf0edadced92dbed653eb1521f7d88b866b801b4dbfd222bd9314aaf51849: Status 404 returned error can't find the container with id c24bf0edadced92dbed653eb1521f7d88b866b801b4dbfd222bd9314aaf51849 Oct 31 00:18:33.411842 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:18:33.411816 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d60169a_68c9_4cfd_bbd7_73ecac659695.slice/crio-9841ac2eaecc67e3dea7f7ac1b0e211b0822c3ae099e21343a3a36f62b636ab6 WatchSource:0}: Error finding container 9841ac2eaecc67e3dea7f7ac1b0e211b0822c3ae099e21343a3a36f62b636ab6: Status 404 returned error can't find the container with id 9841ac2eaecc67e3dea7f7ac1b0e211b0822c3ae099e21343a3a36f62b636ab6 Oct 31 00:18:33.413904 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.413879 2572 kuberuntime_manager.go:1707] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-mhppg_openshift-multus(2d60169a-68c9-4cfd-bbd7-73ecac659695): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 31 00:18:33.413981 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.413918 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-mhppg" podUID="2d60169a-68c9-4cfd-bbd7-73ecac659695" Oct 31 00:18:33.591365 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.591331 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:33.647197 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.647101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:33.647197 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.647185 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:33.647386 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.647266 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:33.647386 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.647294 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:33.647386 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.647343 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs podName:57df0df3-a039-45eb-8ae2-02433f6b0b2e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:34.647326384 +0000 UTC m=+3.127502952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs") pod "network-metrics-daemon-xzjcg" (UID: "57df0df3-a039-45eb-8ae2-02433f6b0b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:33.647386 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.647358 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret podName:a56c984b-c9d9-4d3f-86fd-2314f01a6673 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:34.647351436 +0000 UTC m=+3.127527987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret") pod "global-pull-secret-syncer-z4hfm" (UID: "a56c984b-c9d9-4d3f-86fd-2314f01a6673") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:33.748646 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:33.748605 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:33.748819 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.748774 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:33.748819 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.748800 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:33.748819 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.748814 2572 projected.go:196] Error preparing data for projected volume kube-api-access-chgxt for pod openshift-network-diagnostics/network-check-target-mswcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:33.748999 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:33.748903 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt podName:6249a4c3-cd70-4c06-aa16-9864e6493086 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:34.748883426 +0000 UTC m=+3.229060004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-chgxt" (UniqueName: "kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt") pod "network-check-target-mswcp" (UID: "6249a4c3-cd70-4c06-aa16-9864e6493086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:34.069105 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.068985 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" event={"ID":"2b1af480-ae8f-483e-bd52-947094fcc3ff","Type":"ContainerStarted","Data":"62560de8d575f7f73af9da33245d44cee439e0f9890f3e71b795e6595c2c475d"} Oct 31 00:18:34.073460 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.073395 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:13:33 +0000 UTC" deadline="2028-07-04 08:56:19.433961931 +0000 UTC" Oct 31 00:18:34.073460 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.073460 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14696h37m45.360506498s" Oct 31 00:18:34.090432 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.090319 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b2slc" event={"ID":"b70404c4-e809-42c9-9efc-d6262e52815c","Type":"ContainerStarted","Data":"6c40f3a146031134a5ef8f7edad95480466c4028e385fabc50b348ddf6742a54"} Oct 31 00:18:34.113390 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.113320 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"eab2e57560f114f1accf98cdb298edb1a4f619f6f09723003a23cd02a81f0a33"} Oct 31 00:18:34.137915 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.137877 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bshvl" event={"ID":"1f9f082d-9176-491a-8d5e-281ee6665dae","Type":"ContainerStarted","Data":"5f002c7dbb510dd14f973c08630a802a79cc6ae52e621603bd4d9d78d16292eb"} Oct 31 00:18:34.146738 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.146694 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-npncc" event={"ID":"47350ce3-ba5e-4393-ac0f-653b41bc28fd","Type":"ContainerStarted","Data":"32d47e0e1ce59bc6c4787f4b8e8028b6abf07054417ca8d6c0d4838b95d05581"} Oct 31 00:18:34.149565 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.149460 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" event={"ID":"ad1af2674e0c55384c949adc8f5b6f9c","Type":"ContainerStarted","Data":"f95f9e90756a204a140abb4e56732e931cda0f25c18a8d2ba3d504610a3c90df"} Oct 31 00:18:34.153384 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.153340 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"d3b869eb4ef8b2ca62aee88ee86a131f","Type":"ContainerStarted","Data":"feae4182e7d2d731c263c8b3ddfa1e81e707fef6583ba1cb18aa78ed7682c00f"} Oct 31 00:18:34.158767 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.157810 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhppg" event={"ID":"2d60169a-68c9-4cfd-bbd7-73ecac659695","Type":"ContainerStarted","Data":"9841ac2eaecc67e3dea7f7ac1b0e211b0822c3ae099e21343a3a36f62b636ab6"} Oct 31 00:18:34.162261 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:34.162059 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65a324fc19ef3da53cecb9dc13e0c07696c595996d3c07d1a5dfd427f6a61e6e\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-mhppg" podUID="2d60169a-68c9-4cfd-bbd7-73ecac659695" Oct 31 00:18:34.166048 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.166011 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bfsdv" event={"ID":"3af6b8f1-b029-4ae2-99cb-df0315daa268","Type":"ContainerStarted","Data":"c24bf0edadced92dbed653eb1521f7d88b866b801b4dbfd222bd9314aaf51849"} Oct 31 00:18:34.176016 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.175949 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpxt2" event={"ID":"f99f8918-3e3d-456b-be91-db4120b67424","Type":"ContainerStarted","Data":"e6691222484d14b0c92a573776082782a7b8cfb0674b0154c2ae1a3089d8123e"} Oct 31 00:18:34.179591 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.179521 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gffpv" event={"ID":"05e75b3b-ffb1-4cda-9203-f4882bd385e7","Type":"ContainerStarted","Data":"ff642c15ac0a65be45e23166e5ef3057d41b6735d8489551e522baf6bc44d32e"} Oct 31 00:18:34.180520 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.180451 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:18:34.657703 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.656802 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:34.657703 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.656930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:34.657703 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:34.657090 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:34.657703 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:34.657156 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs podName:57df0df3-a039-45eb-8ae2-02433f6b0b2e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:36.65713678 +0000 UTC m=+5.137313336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs") pod "network-metrics-daemon-xzjcg" (UID: "57df0df3-a039-45eb-8ae2-02433f6b0b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:34.657703 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:34.657610 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:34.657703 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:34.657662 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret podName:a56c984b-c9d9-4d3f-86fd-2314f01a6673 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:36.65764643 +0000 UTC m=+5.137823002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret") pod "global-pull-secret-syncer-z4hfm" (UID: "a56c984b-c9d9-4d3f-86fd-2314f01a6673") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:34.757917 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:34.757876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:34.758240 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:34.758106 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:34.758240 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:34.758134 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:34.758240 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:34.758148 2572 projected.go:196] Error preparing data for projected volume kube-api-access-chgxt for pod openshift-network-diagnostics/network-check-target-mswcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:34.758240 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:34.758215 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt podName:6249a4c3-cd70-4c06-aa16-9864e6493086 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:36.758195034 +0000 UTC m=+5.238371605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-chgxt" (UniqueName: "kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt") pod "network-check-target-mswcp" (UID: "6249a4c3-cd70-4c06-aa16-9864e6493086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:35.057886 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:35.057795 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:35.058057 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:35.057948 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:35.058428 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:35.058406 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:35.058572 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:35.058551 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:35.058653 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:35.058637 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:35.058742 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:35.058722 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:35.073692 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:35.073612 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:13:33 +0000 UTC" deadline="2028-07-12 04:37:27.236031562 +0000 UTC" Oct 31 00:18:35.073692 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:35.073688 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14884h18m52.162347472s" Oct 31 00:18:36.676178 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:36.676050 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:36.676178 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:36.676129 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:36.676738 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:36.676298 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:36.676738 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:36.676366 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs podName:57df0df3-a039-45eb-8ae2-02433f6b0b2e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.676345714 +0000 UTC m=+9.156522271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs") pod "network-metrics-daemon-xzjcg" (UID: "57df0df3-a039-45eb-8ae2-02433f6b0b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:36.676874 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:36.676837 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:36.676935 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:36.676894 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret podName:a56c984b-c9d9-4d3f-86fd-2314f01a6673 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.676877597 +0000 UTC m=+9.157054154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret") pod "global-pull-secret-syncer-z4hfm" (UID: "a56c984b-c9d9-4d3f-86fd-2314f01a6673") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:36.777158 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:36.777069 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:36.777330 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:36.777255 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:36.777330 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:36.777275 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:36.777330 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:36.777288 2572 projected.go:196] Error preparing data for projected volume kube-api-access-chgxt for pod openshift-network-diagnostics/network-check-target-mswcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:36.777449 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:36.777348 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt podName:6249a4c3-cd70-4c06-aa16-9864e6493086 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:40.777330594 +0000 UTC m=+9.257507167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-chgxt" (UniqueName: "kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt") pod "network-check-target-mswcp" (UID: "6249a4c3-cd70-4c06-aa16-9864e6493086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:37.058679 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:37.057781 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:37.058679 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:37.057943 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:37.058679 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:37.058349 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:37.058679 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:37.058445 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:37.058679 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:37.058491 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:37.058679 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:37.058584 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:38.210872 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:38.210326 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bshvl" event={"ID":"1f9f082d-9176-491a-8d5e-281ee6665dae","Type":"ContainerStarted","Data":"20f4311e6f08ff29ebe4bfcfcca1d38e092056f27bf6b6532b431a4bdfb88d64"} Oct 31 00:18:38.213844 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:38.213810 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" event={"ID":"ad1af2674e0c55384c949adc8f5b6f9c","Type":"ContainerStarted","Data":"9cdeeb37adfa2716df0a9041df3a10c4048a6436988ff3b5626b72d131d542c4"} Oct 31 00:18:38.219747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:38.219711 2572 generic.go:356] "Generic (PLEG): container finished" podID="d3b869eb4ef8b2ca62aee88ee86a131f" containerID="0ef232731fce729f61bc4a971a4c314d979c8a05da8d54d6fc7b34cc376a2ff1" exitCode=0 Oct 31 00:18:38.219880 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:38.219767 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"d3b869eb4ef8b2ca62aee88ee86a131f","Type":"ContainerDied","Data":"0ef232731fce729f61bc4a971a4c314d979c8a05da8d54d6fc7b34cc376a2ff1"} Oct 31 00:18:38.314245 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:38.314042 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bshvl" podStartSLOduration=2.039027671 podStartE2EDuration="6.3140062s" totalImagesPullingTime="4.274978529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:38.269602581 +0000 UTC m=+6.749779160" watchObservedRunningTime="2026-10-31 00:18:38.3140062 +0000 UTC m=+6.794182778" Oct 31 00:18:38.360520 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:38.360453 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" podStartSLOduration=6.360431392 podStartE2EDuration="6.360431392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:38.31491586 +0000 UTC m=+6.795092453" watchObservedRunningTime="2026-10-31 00:18:38.360431392 +0000 UTC m=+6.840607969" Oct 31 00:18:39.057691 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:39.057643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:39.057883 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:39.057796 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:39.057951 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:39.057891 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:39.058005 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:39.057982 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:39.058066 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:39.058050 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:39.058156 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:39.058133 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:39.229257 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:39.229220 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"d3b869eb4ef8b2ca62aee88ee86a131f","Type":"ContainerStarted","Data":"28345d3584d1284e590625a134a2e6c1da184bbcc8e5f957f0ffef51a02aac34"} Oct 31 00:18:39.232817 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:39.232749 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpxt2" event={"ID":"f99f8918-3e3d-456b-be91-db4120b67424","Type":"ContainerStarted","Data":"2278f422b1fcf016970553dea4d8f3c229a9d93f92d5aa04b30e69c3127c02c0"} Oct 31 00:18:39.322647 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:39.322520 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" podStartSLOduration=7.322479423 podStartE2EDuration="7.322479423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:39.27694503 +0000 UTC m=+7.757121625" watchObservedRunningTime="2026-10-31 00:18:39.322479423 +0000 UTC m=+7.802656000" Oct 31 00:18:40.191720 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:40.191641 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:40.192690 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:40.192504 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:40.235522 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:40.234693 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jpxt2" podStartSLOduration=2.625807186 podStartE2EDuration="8.234672716s" totalImagesPullingTime="5.60886553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:39.32350516 +0000 UTC m=+7.803681747" watchObservedRunningTime="2026-10-31 00:18:40.234672716 +0000 UTC m=+8.714849289" Oct 31 00:18:40.242097 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:40.241175 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" event={"ID":"2b1af480-ae8f-483e-bd52-947094fcc3ff","Type":"ContainerStarted","Data":"e80f3fa9dcaf4bf63a539dabd16d733e4d70fd87ad0b8265791a22670ec5d3f7"} Oct 31 00:18:40.663562 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:40.663527 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:40.712200 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:40.712162 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:40.712367 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:40.712225 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:40.712367 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:40.712354 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:40.712481 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:40.712413 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs podName:57df0df3-a039-45eb-8ae2-02433f6b0b2e nodeName:}" failed. No retries permitted until 2026-10-31 00:18:48.712396795 +0000 UTC m=+17.192573360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs") pod "network-metrics-daemon-xzjcg" (UID: "57df0df3-a039-45eb-8ae2-02433f6b0b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:40.712823 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:40.712806 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:40.712908 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:40.712851 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret podName:a56c984b-c9d9-4d3f-86fd-2314f01a6673 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:48.712839767 +0000 UTC m=+17.193016333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret") pod "global-pull-secret-syncer-z4hfm" (UID: "a56c984b-c9d9-4d3f-86fd-2314f01a6673") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:40.813286 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:40.813203 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:40.813471 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:40.813415 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:40.813471 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:40.813436 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:40.813471 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:40.813448 2572 projected.go:196] Error preparing data for projected volume kube-api-access-chgxt for pod openshift-network-diagnostics/network-check-target-mswcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:40.813638 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:40.813527 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt podName:6249a4c3-cd70-4c06-aa16-9864e6493086 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:48.813507944 +0000 UTC m=+17.293684499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-chgxt" (UniqueName: "kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt") pod "network-check-target-mswcp" (UID: "6249a4c3-cd70-4c06-aa16-9864e6493086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:40.888073 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:40.888033 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:41.058884 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:41.058026 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:41.058884 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:41.058161 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:41.058884 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:41.058584 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:41.058884 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:41.058698 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:41.058884 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:41.058752 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:41.058884 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:41.058822 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:41.242624 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:41.242542 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:18:41.575771 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:41.575554 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:18:41.993671 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:41.993506 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:18:41.575576521Z","UUID":"5c4c49ea-4075-47f6-95f3-0208c622636b","Handler":null,"Name":"","Endpoint":""} Oct 31 00:18:41.995216 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:41.995077 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:18:41.995216 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:41.995107 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:18:42.246838 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:42.246766 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bfsdv" event={"ID":"3af6b8f1-b029-4ae2-99cb-df0315daa268","Type":"ContainerStarted","Data":"db93ddb156401c43814b59313361ab252178fe5717b02313e29151b8a0dbf46b"} Oct 31 00:18:42.250357 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:42.250321 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" event={"ID":"2b1af480-ae8f-483e-bd52-947094fcc3ff","Type":"ContainerStarted","Data":"0e56c7a4d71b2efa4fb08325f1453bf51747f716ebfa378a5b84402e04c02a95"} Oct 31 00:18:42.251883 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:42.251851 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-npncc" event={"ID":"47350ce3-ba5e-4393-ac0f-653b41bc28fd","Type":"ContainerStarted","Data":"ea673ccf1b87b836ebcc81099310cfeccb29f0b3b95f681aad154e7e636ba6c6"} Oct 31 00:18:42.295930 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:42.295871 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bfsdv" podStartSLOduration=2.079264359 podStartE2EDuration="10.29585258s" totalImagesPullingTime="8.216588221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:42.294907775 +0000 UTC m=+10.775084348" watchObservedRunningTime="2026-10-31 00:18:42.29585258 +0000 UTC m=+10.776029155" Oct 31 00:18:43.057109 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:43.057075 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:43.057109 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:43.057099 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:43.057350 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:43.057088 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:43.057350 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:43.057210 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:43.057350 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:43.057289 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:43.057576 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:43.057371 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:43.256887 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:43.256849 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" event={"ID":"2b1af480-ae8f-483e-bd52-947094fcc3ff","Type":"ContainerStarted","Data":"42eea38f1c5499f5d95fa6016d1c6791f520c025917d306c2da6a48e1c3c6258"} Oct 31 00:18:43.260696 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:43.260660 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b2slc" event={"ID":"b70404c4-e809-42c9-9efc-d6262e52815c","Type":"ContainerStarted","Data":"b3fa5101aeecd52c7cfa582e1fbbe9e0bbe6a535448fd4118dcf06bda46e49c0"} Oct 31 00:18:43.305037 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:43.304980 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-npncc" podStartSLOduration=3.167944529 podStartE2EDuration="11.3049638s" totalImagesPullingTime="8.137019271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:42.3425747 +0000 UTC m=+10.822751277" watchObservedRunningTime="2026-10-31 00:18:43.3049638 +0000 UTC m=+11.785140372" Oct 31 00:18:43.305676 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:43.305643 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" podStartSLOduration=2.52641269 podStartE2EDuration="11.305627718s" totalImagesPullingTime="8.779215028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:43.304430651 +0000 UTC m=+11.784607225" watchObservedRunningTime="2026-10-31 00:18:43.305627718 +0000 UTC m=+11.785804293" Oct 31 00:18:44.183245 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:44.183203 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:44.264765 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:44.264723 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gffpv" event={"ID":"05e75b3b-ffb1-4cda-9203-f4882bd385e7","Type":"ContainerStarted","Data":"b19063a3f666427fe3876914c9b1e90e9a3af1547f5e56abb5aef82f5fbdcfaf"} Oct 31 00:18:44.310840 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:44.310780 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b2slc" podStartSLOduration=4.029501097 podStartE2EDuration="12.310760035s" totalImagesPullingTime="8.281258938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:43.347207953 +0000 UTC m=+11.827384541" watchObservedRunningTime="2026-10-31 00:18:44.310760035 +0000 UTC m=+12.790936609" Oct 31 00:18:44.311487 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:44.311438 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gffpv" podStartSLOduration=2.177249809 podStartE2EDuration="12.311428748s" totalImagesPullingTime="10.134178939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:44.310467114 +0000 UTC m=+12.790643678" watchObservedRunningTime="2026-10-31 00:18:44.311428748 +0000 UTC m=+12.791605325" Oct 31 00:18:44.366606 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:44.366568 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:45.057302 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:45.057256 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:45.057302 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:45.057283 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:45.057570 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:45.057266 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:45.057570 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:45.057395 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:45.057570 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:45.057509 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:45.057725 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:45.057576 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:46.417836 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:46.417634 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:46.563835 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:46.563796 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:46.816864 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:46.816797 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:46.963328 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:46.963276 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:47.057862 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:47.057832 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:47.058049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:47.057951 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:47.058049 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:47.057970 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:47.058049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:47.057984 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:47.058215 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:47.058077 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:47.058215 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:47.058172 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:47.107983 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:47.107889 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:47.907722 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:47.907672 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:48.097694 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:48.097662 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:48.275898 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:48.275868 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"eedb5073e0ea0c851e399952ddb84421cd92c5ee6281b0a40d3bad25951455ba"} Oct 31 00:18:48.275989 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:48.275961 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"77da00772a3a2f9139a5ed71bf533f6a20a20bcf88af907cf7f79ec4a442c0dd"} Oct 31 00:18:48.776154 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:48.775932 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:48.776154 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:48.776141 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:48.776334 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:48.776067 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:48.776334 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:48.776227 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret podName:a56c984b-c9d9-4d3f-86fd-2314f01a6673 nodeName:}" failed. No retries permitted until 2026-10-31 00:19:04.776212898 +0000 UTC m=+33.256389476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret") pod "global-pull-secret-syncer-z4hfm" (UID: "a56c984b-c9d9-4d3f-86fd-2314f01a6673") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:18:48.776334 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:48.776242 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:48.776334 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:48.776284 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs podName:57df0df3-a039-45eb-8ae2-02433f6b0b2e nodeName:}" failed. No retries permitted until 2026-10-31 00:19:04.776269686 +0000 UTC m=+33.256446240 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs") pod "network-metrics-daemon-xzjcg" (UID: "57df0df3-a039-45eb-8ae2-02433f6b0b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:18:48.876798 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:48.876769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:48.876960 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:48.876901 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:18:48.876960 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:48.876914 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:18:48.876960 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:48.876923 2572 projected.go:196] Error preparing data for projected volume kube-api-access-chgxt for pod openshift-network-diagnostics/network-check-target-mswcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:48.877063 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:48.876971 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt podName:6249a4c3-cd70-4c06-aa16-9864e6493086 nodeName:}" failed. No retries permitted until 2026-10-31 00:19:04.876955995 +0000 UTC m=+33.357132559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-chgxt" (UniqueName: "kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt") pod "network-check-target-mswcp" (UID: "6249a4c3-cd70-4c06-aa16-9864e6493086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:18:49.021370 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.021326 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:49.057352 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.057273 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:49.057352 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.057291 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:49.057352 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.057305 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:49.057602 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:49.057380 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:49.057602 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:49.057525 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:49.057751 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:49.057733 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:49.252713 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.252674 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:49.280369 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.280348 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:18:49.280702 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.280684 2572 generic.go:356] "Generic (PLEG): container finished" podID="064705a9-54fd-4528-ae88-1cb4596c5b6e" containerID="eedb5073e0ea0c851e399952ddb84421cd92c5ee6281b0a40d3bad25951455ba" exitCode=1 Oct 31 00:18:49.280776 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.280735 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerDied","Data":"eedb5073e0ea0c851e399952ddb84421cd92c5ee6281b0a40d3bad25951455ba"} Oct 31 00:18:49.280813 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.280786 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"37f3754304ecc7385949096997d58a3c12f2d8bd51b8be34336136622e02f02e"} Oct 31 00:18:49.280844 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.280823 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"8833a1797703de4932e785b367ea6b39aec2d081cc45838acb01562889db722e"} Oct 31 00:18:49.280872 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.280852 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"3512227f4936aee66bf31fbd507d4f31f6f22b85adc9df50d7356788dbd9ee75"} Oct 31 00:18:49.280899 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.280879 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"160ca2edb2757a9aecd660a111263850c33eac8c57835c15988ed0bc9153a51c"} Oct 31 00:18:49.981815 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:49.981784 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-npncc_47350ce3-ba5e-4393-ac0f-653b41bc28fd/dns-node-resolver/0.log" Oct 31 00:18:50.137437 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:50.137403 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:50.284680 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:50.284598 2572 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="2080f43e4022d6e03623d14b050bd3e71ef95da20a1de21e8b1e2f7f52cc937a" exitCode=0 Oct 31 00:18:50.284680 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:50.284659 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhppg" event={"ID":"2d60169a-68c9-4cfd-bbd7-73ecac659695","Type":"ContainerDied","Data":"2080f43e4022d6e03623d14b050bd3e71ef95da20a1de21e8b1e2f7f52cc937a"} Oct 31 00:18:50.307048 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:50.307014 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:50.567099 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:50.567011 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:50.773843 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:50.773812 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jpxt2_f99f8918-3e3d-456b-be91-db4120b67424/node-ca/0.log" Oct 31 00:18:51.057629 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:51.057538 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:51.057629 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:51.057595 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:51.057839 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:51.057610 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:51.057839 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:51.057707 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:51.057943 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:51.057857 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:51.057996 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:51.057963 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:51.291371 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:51.291342 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:18:51.291949 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:51.291766 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"0a478e91766e245d33373432259a862523c249dde699de0b708b357ee615c519"} Oct 31 00:18:51.967793 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:51.967755 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:52.296438 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:52.296371 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhppg" event={"ID":"2d60169a-68c9-4cfd-bbd7-73ecac659695","Type":"ContainerStarted","Data":"85eb71e41ab9d186337a379ac56c4ea08c2969d56d8bf1fcf2acacf634b0637f"} Oct 31 00:18:53.057318 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.057080 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:53.057484 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.057142 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:53.057484 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:53.057365 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:53.057484 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.057151 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:53.057583 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:53.057419 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:53.057583 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:53.057534 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:53.161240 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.161205 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:53.300338 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.300302 2572 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="85eb71e41ab9d186337a379ac56c4ea08c2969d56d8bf1fcf2acacf634b0637f" exitCode=0 Oct 31 00:18:53.301018 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.300377 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhppg" event={"ID":"2d60169a-68c9-4cfd-bbd7-73ecac659695","Type":"ContainerDied","Data":"85eb71e41ab9d186337a379ac56c4ea08c2969d56d8bf1fcf2acacf634b0637f"} Oct 31 00:18:53.304049 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.304030 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:18:53.304370 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.304348 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"97e096e8b9474ffb15cf89d1d949b6066fc69c3700f620f2a16c57f3f0eac0a9"} Oct 31 00:18:53.304647 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.304632 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:53.304758 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.304744 2572 scope.go:122] "RemoveContainer" containerID="eedb5073e0ea0c851e399952ddb84421cd92c5ee6281b0a40d3bad25951455ba" Oct 31 00:18:53.321614 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.321589 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:53.395354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.395312 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:53.608113 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:53.608034 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:54.027476 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.027431 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:54.309151 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.309067 2572 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="75204a122fd669d223dc818da5588c4c345e1b1d953f79be19a38adb3eb5ee42" exitCode=0 Oct 31 00:18:54.309151 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.309132 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhppg" event={"ID":"2d60169a-68c9-4cfd-bbd7-73ecac659695","Type":"ContainerDied","Data":"75204a122fd669d223dc818da5588c4c345e1b1d953f79be19a38adb3eb5ee42"} Oct 31 00:18:54.316135 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.316115 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:18:54.316477 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.316439 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" event={"ID":"064705a9-54fd-4528-ae88-1cb4596c5b6e","Type":"ContainerStarted","Data":"17fe04c87a624f13fcdb8ecac525f37a540cac16e3151a5192b9f99d0757fda4"} Oct 31 00:18:54.316579 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.316529 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:18:54.316760 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.316744 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:54.332264 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.332233 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:18:54.418387 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.418334 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" podStartSLOduration=7.720601434 podStartE2EDuration="22.418318318s" totalImagesPullingTime="14.697716884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:54.418182388 +0000 UTC m=+22.898358973" watchObservedRunningTime="2026-10-31 00:18:54.418318318 +0000 UTC m=+22.898494890" Oct 31 00:18:54.442639 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.442607 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:54.614653 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.614555 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:54.868698 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:54.868674 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:55.057591 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.057551 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:55.057591 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.057580 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:55.057837 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.057568 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:55.057837 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:55.057675 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:55.057837 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:55.057768 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:55.057956 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:55.057854 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:55.070435 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.070405 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:55.215144 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.215059 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:55.321069 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.321039 2572 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="619c0b2273b3273804962b179ab5ba39603c66368fb711a5855e4f46632e38e6" exitCode=0 Oct 31 00:18:55.321522 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.321105 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhppg" event={"ID":"2d60169a-68c9-4cfd-bbd7-73ecac659695","Type":"ContainerDied","Data":"619c0b2273b3273804962b179ab5ba39603c66368fb711a5855e4f46632e38e6"} Oct 31 00:18:55.321522 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.321182 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:18:55.383695 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.383660 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:55.589859 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.589769 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:55.812570 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.812507 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:55.984471 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:55.984351 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:56.324061 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:56.323982 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:18:56.612364 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:56.612333 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.008760 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.008672 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.057159 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.057128 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:57.057159 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.057150 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:57.057309 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.057161 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:57.057309 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:57.057251 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:57.057382 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:57.057353 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:57.057427 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:57.057415 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:57.188599 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.188569 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.328730 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.328654 2572 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="becf88363f160ddc4d838ca7a88728489a7638a8c8485bc416e3fdefb90ff936" exitCode=0 Oct 31 00:18:57.328730 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.328691 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhppg" event={"ID":"2d60169a-68c9-4cfd-bbd7-73ecac659695","Type":"ContainerDied","Data":"becf88363f160ddc4d838ca7a88728489a7638a8c8485bc416e3fdefb90ff936"} Oct 31 00:18:57.331852 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.331825 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.478926 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.478893 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.814320 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:57.814230 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:58.160754 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:58.160667 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:58.160942 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:58.160803 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:18:58.161280 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:58.161256 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:18:58.337174 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:58.337138 2572 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="c1c6d4c2f0073098bfee4185dc530e40c35e8378c08223303414dd8486ca4f6e" exitCode=0 Oct 31 00:18:58.337559 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:58.337183 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhppg" event={"ID":"2d60169a-68c9-4cfd-bbd7-73ecac659695","Type":"ContainerDied","Data":"c1c6d4c2f0073098bfee4185dc530e40c35e8378c08223303414dd8486ca4f6e"} Oct 31 00:18:58.421596 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:58.421561 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:59.007385 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:59.007346 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:59.057365 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:59.057330 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:18:59.057365 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:59.057366 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:18:59.057648 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:59.057366 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:18:59.057648 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:59.057485 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:18:59.057648 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:59.057617 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:18:59.057785 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:18:59.057690 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:18:59.233346 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:59.233301 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:59.343108 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:59.343031 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhppg" event={"ID":"2d60169a-68c9-4cfd-bbd7-73ecac659695","Type":"ContainerStarted","Data":"e962c4c9048412b253c2a34fbfcd40af5da446d6f1c7c56f1720035c87f31033"} Oct 31 00:18:59.622706 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:59.622616 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:59.805323 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:18:59.805287 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:00.396516 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:00.396477 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.012975 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.012939 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.057928 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.057896 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:01.057928 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.057923 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:01.058123 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.057995 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:01.058123 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:01.058010 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:01.058123 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:01.058067 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:01.058219 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:01.058144 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:01.205437 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.205406 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.534329 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.534301 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:01.534773 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.534571 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:19:01.551597 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.551541 2572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" podUID="064705a9-54fd-4528-ae88-1cb4596c5b6e" containerName="ovnkube-controller" probeResult="failure" output="" Oct 31 00:19:01.560909 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.560869 2572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" podUID="064705a9-54fd-4528-ae88-1cb4596c5b6e" containerName="ovnkube-controller" probeResult="failure" output="" Oct 31 00:19:01.600035 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.600003 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.793919 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:01.793823 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:02.046125 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:02.046042 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:02.196571 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:02.196533 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:02.766283 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:02.766251 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:03.011558 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:03.011520 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:03.057649 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:03.057563 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:03.057649 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:03.057594 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:03.057649 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:03.057601 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:03.057855 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:03.057675 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:03.057855 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:03.057724 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:03.057855 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:03.057810 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:03.202991 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:03.202959 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:03.416345 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:03.416264 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:03.576398 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:03.576367 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:03.723493 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:03.723385 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:03.869344 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:03.869314 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:04.039063 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:04.038984 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:04.461879 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:04.461799 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:04.606791 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:04.606761 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:04.800855 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:04.800776 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:04.800855 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:04.800837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:04.801036 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:04.800934 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:19:04.801036 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:04.800940 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:19:04.801036 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:04.800999 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret podName:a56c984b-c9d9-4d3f-86fd-2314f01a6673 nodeName:}" failed. No retries permitted until 2026-10-31 00:19:36.800985134 +0000 UTC m=+65.281161700 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret") pod "global-pull-secret-syncer-z4hfm" (UID: "a56c984b-c9d9-4d3f-86fd-2314f01a6673") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:19:04.801036 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:04.801013 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs podName:57df0df3-a039-45eb-8ae2-02433f6b0b2e nodeName:}" failed. No retries permitted until 2026-10-31 00:19:36.801007047 +0000 UTC m=+65.281183611 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs") pod "network-metrics-daemon-xzjcg" (UID: "57df0df3-a039-45eb-8ae2-02433f6b0b2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:19:04.902042 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:04.902010 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:04.902433 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:04.902166 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:19:04.902433 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:04.902186 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:19:04.902433 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:04.902196 2572 projected.go:196] Error preparing data for projected volume kube-api-access-chgxt for pod openshift-network-diagnostics/network-check-target-mswcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:19:04.902433 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:04.902248 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt podName:6249a4c3-cd70-4c06-aa16-9864e6493086 nodeName:}" failed. No retries permitted until 2026-10-31 00:19:36.902233521 +0000 UTC m=+65.382410078 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-chgxt" (UniqueName: "kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt") pod "network-check-target-mswcp" (UID: "6249a4c3-cd70-4c06-aa16-9864e6493086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:19:05.057611 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:05.057545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:05.057820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:05.057644 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:05.057820 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:05.057659 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:05.057820 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:05.057717 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:05.057820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:05.057730 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:05.057820 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:05.057813 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:06.444472 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:06.444413 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:06.644534 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:06.644490 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:06.890254 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:06.890167 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:07.057781 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:07.057738 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:07.057781 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:07.057766 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:07.058040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:07.057740 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:07.058040 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:07.057869 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:07.058040 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:07.057929 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:07.058040 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:07.057971 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:07.090736 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:07.090702 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:07.263099 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:07.263010 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:07.446415 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:07.446383 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:08.799937 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:08.799900 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:08.941540 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:08.941503 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:09.057941 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:09.057861 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:09.058087 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:09.057860 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:09.058087 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:09.057975 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:09.058087 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:09.057863 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:09.058185 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:09.058035 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:09.058185 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:09.058125 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:11.057649 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:11.057616 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:11.058042 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:11.057680 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:11.058042 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:11.057702 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:11.058042 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:11.057778 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:11.058042 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:11.057818 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:11.058042 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:11.057891 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:11.888792 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:11.888739 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mhppg" podStartSLOduration=15.971539395 podStartE2EDuration="39.888724564s" totalImagesPullingTime="23.917185169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:59.420254249 +0000 UTC m=+27.900430831" watchObservedRunningTime="2026-10-31 00:19:11.888724564 +0000 UTC m=+40.368901175" Oct 31 00:19:12.096033 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:12.096003 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:12.368366 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:12.368333 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:12.579831 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:12.579791 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:12.799882 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:12.799789 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:13.058019 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:13.057930 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:13.058019 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:13.057952 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:13.058019 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:13.057958 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:13.058281 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:13.058060 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:13.058281 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:13.058196 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:13.058281 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:13.058272 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:13.940353 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:13.940315 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z4hfm"] Oct 31 00:19:13.940734 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:13.940444 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:13.940734 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:13.940574 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:13.944672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:13.944641 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mswcp"] Oct 31 00:19:13.944826 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:13.944755 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:13.944893 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:13.944853 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:13.945659 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:13.945636 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xzjcg"] Oct 31 00:19:13.945779 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:13.945744 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:13.945882 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:13.945855 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:15.605933 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:15.605740 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:15.773822 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:15.773784 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:15.945341 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:15.945260 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:16.057677 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:16.057643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:16.057842 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:16.057685 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:16.057898 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:16.057834 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:16.057898 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:16.057877 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:16.058017 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:16.057995 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:16.058106 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:16.058089 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:16.141756 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:16.141718 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:17.568986 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:17.568951 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:17.711943 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:17.711909 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:17.910441 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:17.910357 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:18.058103 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.058070 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:18.058248 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.058070 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:18.058248 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:18.058189 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mswcp" podUID="6249a4c3-cd70-4c06-aa16-9864e6493086" Oct 31 00:19:18.058317 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:18.058272 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzjcg" podUID="57df0df3-a039-45eb-8ae2-02433f6b0b2e" Oct 31 00:19:18.058317 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.058070 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:18.058381 ip-10-0-129-199 kubenswrapper[2572]: E1031 00:19:18.058353 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4hfm" podUID="a56c984b-c9d9-4d3f-86fd-2314f01a6673" Oct 31 00:19:18.276209 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.276127 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeReady" Oct 31 00:19:18.276378 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.276250 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:19:18.353508 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.353477 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nqpmv"] Oct 31 00:19:18.355340 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.355324 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.376226 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.376192 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kqx7w"] Oct 31 00:19:18.378052 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.378034 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:18.378382 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.378349 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:19:18.378657 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.378640 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rltkk\"" Oct 31 00:19:18.379095 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.379078 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:19:18.385952 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.385927 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nqpmv"] Oct 31 00:19:18.405549 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.405518 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kqx7w"] Oct 31 00:19:18.437471 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.437428 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:19:18.437471 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.437432 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:19:18.478051 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.478018 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qg2xf"] Oct 31 00:19:18.479919 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.479902 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.485385 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.485362 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l44c8\"" Oct 31 00:19:18.485556 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.485443 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:19:18.498569 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.498534 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:19:18.498741 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.498598 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:19:18.498741 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.498622 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:19:18.502936 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.502904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f8e29e-fb86-4153-b168-0bb1743e44cd-config-volume\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.503060 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.502940 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f8e29e-fb86-4153-b168-0bb1743e44cd-tmp-dir\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.503060 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.502968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kl9k\" (UniqueName: \"kubernetes.io/projected/d6f8e29e-fb86-4153-b168-0bb1743e44cd-kube-api-access-7kl9k\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.503161 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.503078 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f8e29e-fb86-4153-b168-0bb1743e44cd-metrics-tls\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.503161 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.503128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkrbb\" (UniqueName: \"kubernetes.io/projected/7dd2bec9-6796-4e26-ad19-bfd15842852b-kube-api-access-nkrbb\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:18.503267 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.503248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dd2bec9-6796-4e26-ad19-bfd15842852b-cert\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:18.504900 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.504874 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qg2xf"] Oct 31 00:19:18.582265 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.582235 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:19:18.582672 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.582445 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5c7xv\"" Oct 31 00:19:18.603761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.603719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abe218cc-f275-4474-a54e-3ba513e969c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.603761 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.603765 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khv2q\" (UniqueName: \"kubernetes.io/projected/abe218cc-f275-4474-a54e-3ba513e969c2-kube-api-access-khv2q\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.604011 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.603871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrbb\" (UniqueName: \"kubernetes.io/projected/7dd2bec9-6796-4e26-ad19-bfd15842852b-kube-api-access-nkrbb\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:18.604011 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.603916 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abe218cc-f275-4474-a54e-3ba513e969c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.604011 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.603952 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abe218cc-f275-4474-a54e-3ba513e969c2-crio-socket\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.604011 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.604000 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dd2bec9-6796-4e26-ad19-bfd15842852b-cert\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:18.604194 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.604089 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f8e29e-fb86-4153-b168-0bb1743e44cd-config-volume\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.604194 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.604123 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f8e29e-fb86-4153-b168-0bb1743e44cd-tmp-dir\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.604194 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.604148 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl9k\" (UniqueName: \"kubernetes.io/projected/d6f8e29e-fb86-4153-b168-0bb1743e44cd-kube-api-access-7kl9k\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.604194 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.604182 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f8e29e-fb86-4153-b168-0bb1743e44cd-metrics-tls\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.604386 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.604205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abe218cc-f275-4474-a54e-3ba513e969c2-data-volume\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.604551 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.604533 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dcdc4b64f-p6xjm"] Oct 31 00:19:18.604551 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.604538 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f8e29e-fb86-4153-b168-0bb1743e44cd-tmp-dir\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.604745 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.604726 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f8e29e-fb86-4153-b168-0bb1743e44cd-config-volume\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.606387 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.606373 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.608364 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.608335 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f8e29e-fb86-4153-b168-0bb1743e44cd-metrics-tls\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.608477 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.608372 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dd2bec9-6796-4e26-ad19-bfd15842852b-cert\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:18.631637 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.631606 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcdc4b64f-p6xjm"] Oct 31 00:19:18.634148 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.634118 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrbb\" (UniqueName: \"kubernetes.io/projected/7dd2bec9-6796-4e26-ad19-bfd15842852b-kube-api-access-nkrbb\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:18.635370 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.635314 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:19:18.635630 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.635608 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:19:18.635702 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.635664 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:19:18.635797 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.635784 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:19:18.635839 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.635786 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:19:18.641716 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.641697 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kl9k\" (UniqueName: \"kubernetes.io/projected/d6f8e29e-fb86-4153-b168-0bb1743e44cd-kube-api-access-7kl9k\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.669534 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.669502 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:18.672105 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.672079 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:18.679182 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.679159 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p7sw8\"" Oct 31 00:19:18.679182 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.679172 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:19:18.679376 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.679238 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:19:18.688536 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.688504 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:18.704967 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.704935 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abe218cc-f275-4474-a54e-3ba513e969c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.705134 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705082 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.705195 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnvmn\" (UniqueName: \"kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.705248 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abe218cc-f275-4474-a54e-3ba513e969c2-crio-socket\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.705299 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.705299 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705282 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.705393 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705346 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abe218cc-f275-4474-a54e-3ba513e969c2-crio-socket\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.705393 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705382 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.705507 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705433 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.705605 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705581 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abe218cc-f275-4474-a54e-3ba513e969c2-data-volume\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.705661 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abe218cc-f275-4474-a54e-3ba513e969c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.705716 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705670 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khv2q\" (UniqueName: \"kubernetes.io/projected/abe218cc-f275-4474-a54e-3ba513e969c2-kube-api-access-khv2q\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.705716 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705695 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.705926 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.705902 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abe218cc-f275-4474-a54e-3ba513e969c2-data-volume\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.706144 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.706121 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abe218cc-f275-4474-a54e-3ba513e969c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.707529 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.707509 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abe218cc-f275-4474-a54e-3ba513e969c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.740154 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.740129 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khv2q\" (UniqueName: \"kubernetes.io/projected/abe218cc-f275-4474-a54e-3ba513e969c2-kube-api-access-khv2q\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.791576 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.791545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:18.806675 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.806646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.806675 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.806679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnvmn\" (UniqueName: \"kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.806871 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.806700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.806871 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.806716 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.806871 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.806741 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.806871 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.806777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.806871 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.806830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.807113 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.807097 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.807549 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.807527 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.807641 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.807563 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.807688 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.807656 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.809173 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.809155 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.809533 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.809519 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.843416 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.843387 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnvmn\" (UniqueName: \"kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.921956 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.921913 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:18.930930 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.930892 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:18.999278 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:18.999220 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nqpmv"] Oct 31 00:19:18.999603 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:18.999574 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd2bec9_6796_4e26_ad19_bfd15842852b.slice/crio-dac696112bcd1e84da4742935af48bb50d04e30f60e0d9a417e721be6ef95fb3 WatchSource:0}: Error finding container dac696112bcd1e84da4742935af48bb50d04e30f60e0d9a417e721be6ef95fb3: Status 404 returned error can't find the container with id dac696112bcd1e84da4742935af48bb50d04e30f60e0d9a417e721be6ef95fb3 Oct 31 00:19:19.000259 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.000236 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kqx7w"] Oct 31 00:19:19.000545 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:19.000517 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f8e29e_fb86_4153_b168_0bb1743e44cd.slice/crio-14346ce38b021388565ff9c29e76dd9908641220894197e95fe63cd377c093d8 WatchSource:0}: Error finding container 14346ce38b021388565ff9c29e76dd9908641220894197e95fe63cd377c093d8: Status 404 returned error can't find the container with id 14346ce38b021388565ff9c29e76dd9908641220894197e95fe63cd377c093d8 Oct 31 00:19:19.087126 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.087073 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qg2xf"] Oct 31 00:19:19.088220 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:19.088193 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe218cc_f275_4474_a54e_3ba513e969c2.slice/crio-89377edc5335c112e15939e07830a6b41884d0d911ca703e93dc72dd7fd502cc WatchSource:0}: Error finding container 89377edc5335c112e15939e07830a6b41884d0d911ca703e93dc72dd7fd502cc: Status 404 returned error can't find the container with id 89377edc5335c112e15939e07830a6b41884d0d911ca703e93dc72dd7fd502cc Oct 31 00:19:19.124364 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.124278 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:19.219291 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.219270 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcdc4b64f-p6xjm"] Oct 31 00:19:19.220677 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:19.220651 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa8cd46_8095_4e93_b23a_0a422a47eab3.slice/crio-606f1de1f43b9934f13b8fe518dffe1d71692f202bbbc2d49e76ef0e988e8bbf WatchSource:0}: Error finding container 606f1de1f43b9934f13b8fe518dffe1d71692f202bbbc2d49e76ef0e988e8bbf: Status 404 returned error can't find the container with id 606f1de1f43b9934f13b8fe518dffe1d71692f202bbbc2d49e76ef0e988e8bbf Oct 31 00:19:19.389928 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.389844 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcdc4b64f-p6xjm" event={"ID":"0fa8cd46-8095-4e93-b23a-0a422a47eab3","Type":"ContainerStarted","Data":"606f1de1f43b9934f13b8fe518dffe1d71692f202bbbc2d49e76ef0e988e8bbf"} Oct 31 00:19:19.391554 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.391519 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kqx7w" event={"ID":"7dd2bec9-6796-4e26-ad19-bfd15842852b","Type":"ContainerStarted","Data":"dac696112bcd1e84da4742935af48bb50d04e30f60e0d9a417e721be6ef95fb3"} Oct 31 00:19:19.393404 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.393369 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qg2xf" event={"ID":"abe218cc-f275-4474-a54e-3ba513e969c2","Type":"ContainerStarted","Data":"6d55381a9ad0c7f31a7501e366d7feb34a7bd3022c6920da9d2f6f0cd713cb7c"} Oct 31 00:19:19.393540 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.393428 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qg2xf" event={"ID":"abe218cc-f275-4474-a54e-3ba513e969c2","Type":"ContainerStarted","Data":"89377edc5335c112e15939e07830a6b41884d0d911ca703e93dc72dd7fd502cc"} Oct 31 00:19:19.394987 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.394956 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nqpmv" event={"ID":"d6f8e29e-fb86-4153-b168-0bb1743e44cd","Type":"ContainerStarted","Data":"14346ce38b021388565ff9c29e76dd9908641220894197e95fe63cd377c093d8"} Oct 31 00:19:19.545994 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.545959 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:19.704174 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:19.704076 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:20.057762 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.057713 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:20.057762 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.057736 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:20.057938 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.057814 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:20.075743 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.075715 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:19:20.088084 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.088056 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:19:20.088771 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.088727 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9wq7f\"" Oct 31 00:19:20.088963 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.088947 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dsrzd\"" Oct 31 00:19:20.089339 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.089322 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:19:20.101154 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.101111 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:19:20.101407 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.101386 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6b92m\"" Oct 31 00:19:20.400446 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:20.400404 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qg2xf" event={"ID":"abe218cc-f275-4474-a54e-3ba513e969c2","Type":"ContainerStarted","Data":"be92df30b194754060bc4e728d15956282c01affdfb1225e8745fb7e188d3672"} Oct 31 00:19:21.405598 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.405560 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nqpmv" event={"ID":"d6f8e29e-fb86-4153-b168-0bb1743e44cd","Type":"ContainerStarted","Data":"e8f5a5ffd3664d13de029c4dc7487cea4ddcee60fe589e2585ed90077728e1f2"} Oct 31 00:19:21.406059 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.405624 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nqpmv" event={"ID":"d6f8e29e-fb86-4153-b168-0bb1743e44cd","Type":"ContainerStarted","Data":"d051ad47b1478ce65e38b345a011212ee445793cf4967d9469e56c18d4612854"} Oct 31 00:19:21.406059 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.405853 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:21.441247 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.441208 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69b4c98b6b-w7lch"] Oct 31 00:19:21.444420 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.444073 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.466766 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.466721 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-nqpmv" podStartSLOduration=1.517266773 podStartE2EDuration="3.466707172s" totalImagesPullingTime="1.949440399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:21.466227786 +0000 UTC m=+49.946404359" watchObservedRunningTime="2026-10-31 00:19:21.466707172 +0000 UTC m=+49.946883745" Oct 31 00:19:21.485691 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.485662 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:21.498282 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.498248 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 00:19:21.499309 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.499285 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69b4c98b6b-w7lch"] Oct 31 00:19:21.630948 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.630912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.631151 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.631019 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.631151 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.631061 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.631151 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.631086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzj5s\" (UniqueName: \"kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.631151 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.631136 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.631360 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.631161 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.631360 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.631250 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.631360 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.631292 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.690646 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.690560 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:21.732047 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.732006 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.732047 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.732046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.732294 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.732130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.732294 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.732163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.732294 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.732188 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.732294 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.732214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.732294 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.732234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.732294 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.732259 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzj5s\" (UniqueName: \"kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.734062 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.734029 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.734371 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.734352 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.734658 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.734636 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.734778 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.734747 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.735794 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.735768 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.735947 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.735926 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.738635 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.738608 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.768859 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.768816 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzj5s\" (UniqueName: \"kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:21.774371 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.774340 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf"] Oct 31 00:19:21.777929 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.777906 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:21.796747 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.796716 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:19:21.797001 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.796981 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:19:21.797090 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.797017 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:19:21.800033 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.799998 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:19:21.804006 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.803978 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf"] Oct 31 00:19:21.804425 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.804204 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:19:21.804425 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.804387 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f62jd\"" Oct 31 00:19:21.839178 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.839153 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf"] Oct 31 00:19:21.842063 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.842040 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:21.865756 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.865729 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lhrff"] Oct 31 00:19:21.868188 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.868163 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:21.869190 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.869164 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf"] Oct 31 00:19:21.874400 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.874323 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:21.899032 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.898997 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:19:21.899211 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.899193 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:19:21.900300 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.900282 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:19:21.900434 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.900368 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:19:21.900679 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.900663 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:19:21.900741 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.900709 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pk56l\"" Oct 31 00:19:21.900778 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.900751 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v4wl8\"" Oct 31 00:19:21.900815 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.900801 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:19:21.934268 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934227 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:21.934440 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934283 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:21.934440 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934313 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2pd\" (UniqueName: \"kubernetes.io/projected/364a8716-433d-4181-94ed-f6ccd33a0828-kube-api-access-6m2pd\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:21.934440 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:21.934440 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934390 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:21.934440 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934410 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a38f09d-12b6-481c-bb13-c686226854a6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:21.934701 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934492 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:21.934701 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934517 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/364a8716-433d-4181-94ed-f6ccd33a0828-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:21.934701 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934542 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:21.934701 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:21.934588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drh6b\" (UniqueName: \"kubernetes.io/projected/8a38f09d-12b6-481c-bb13-c686226854a6-kube-api-access-drh6b\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.035088 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035056 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drh6b\" (UniqueName: \"kubernetes.io/projected/8a38f09d-12b6-481c-bb13-c686226854a6-kube-api-access-drh6b\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.035155 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-root\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.035155 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035127 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.035261 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035157 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-wtmp\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.035261 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035199 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.035261 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035228 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-textfile\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.035261 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035257 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.035435 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035278 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2pd\" (UniqueName: \"kubernetes.io/projected/364a8716-433d-4181-94ed-f6ccd33a0828-kube-api-access-6m2pd\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.035435 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035305 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.035435 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.035435 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035342 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a38f09d-12b6-481c-bb13-c686226854a6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.035435 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035374 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.035435 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035409 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwv2b\" (UniqueName: \"kubernetes.io/projected/465a3984-adbc-4677-b22f-9e9240a770f8-kube-api-access-rwv2b\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.035435 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.035779 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035473 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/364a8716-433d-4181-94ed-f6ccd33a0828-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.035779 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.035779 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-metrics-client-ca\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.035779 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-sys\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.035779 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.035598 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-tls\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.036284 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.036262 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.039194 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.037237 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/364a8716-433d-4181-94ed-f6ccd33a0828-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.039194 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.037588 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.039194 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.038283 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a38f09d-12b6-481c-bb13-c686226854a6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.039194 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.038514 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.039194 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.039119 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:22.041994 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.041965 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.042130 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.041971 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.043377 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.043336 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.058032 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.058000 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:22.066429 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.066404 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drh6b\" (UniqueName: \"kubernetes.io/projected/8a38f09d-12b6-481c-bb13-c686226854a6-kube-api-access-drh6b\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.066569 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.066546 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2pd\" (UniqueName: \"kubernetes.io/projected/364a8716-433d-4181-94ed-f6ccd33a0828-kube-api-access-6m2pd\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.094088 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.094055 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:22.136778 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.136719 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.136987 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.136786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwv2b\" (UniqueName: \"kubernetes.io/projected/465a3984-adbc-4677-b22f-9e9240a770f8-kube-api-access-rwv2b\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.136987 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.136832 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-metrics-client-ca\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.136987 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.136875 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-sys\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.136987 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.136898 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-tls\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.136987 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.136921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-root\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.136987 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.136948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.136987 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.136987 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-wtmp\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.137425 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.137034 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-textfile\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.137425 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.137388 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-textfile\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.137571 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.137438 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.137571 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.137481 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-root\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.137702 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.137584 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-sys\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.137702 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.137601 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-wtmp\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.137951 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.137927 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-metrics-client-ca\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.141890 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.139905 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.144679 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.144642 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-tls\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.164524 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.164405 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwv2b\" (UniqueName: \"kubernetes.io/projected/465a3984-adbc-4677-b22f-9e9240a770f8-kube-api-access-rwv2b\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.178573 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.178347 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:22.195092 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.195058 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:22.218660 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:22.216931 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465a3984_adbc_4677_b22f_9e9240a770f8.slice/crio-88b0dbdd3eeacc6d7a65b3c38121d08f26c12b62eb27287c394adbbf1af68fc5 WatchSource:0}: Error finding container 88b0dbdd3eeacc6d7a65b3c38121d08f26c12b62eb27287c394adbbf1af68fc5: Status 404 returned error can't find the container with id 88b0dbdd3eeacc6d7a65b3c38121d08f26c12b62eb27287c394adbbf1af68fc5 Oct 31 00:19:22.226510 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.226446 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:22.366872 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.366847 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69b4c98b6b-w7lch"] Oct 31 00:19:22.368168 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:22.368138 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dae54b3_dceb_4795_85ec_42f9fb9ff256.slice/crio-c4ab2cfa8ceb7546c2058faf373f9b324a7aafcaf5d65b102adbdd9bd438e86f WatchSource:0}: Error finding container c4ab2cfa8ceb7546c2058faf373f9b324a7aafcaf5d65b102adbdd9bd438e86f: Status 404 returned error can't find the container with id c4ab2cfa8ceb7546c2058faf373f9b324a7aafcaf5d65b102adbdd9bd438e86f Oct 31 00:19:22.408990 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.408866 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:22.411520 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.411486 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kqx7w" event={"ID":"7dd2bec9-6796-4e26-ad19-bfd15842852b","Type":"ContainerStarted","Data":"3913e12d8a1548ea4fa1f4c526bf7ed771d8861178ea79190227475f0dadcfa3"} Oct 31 00:19:22.413226 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.413190 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lhrff" event={"ID":"465a3984-adbc-4677-b22f-9e9240a770f8","Type":"ContainerStarted","Data":"88b0dbdd3eeacc6d7a65b3c38121d08f26c12b62eb27287c394adbbf1af68fc5"} Oct 31 00:19:22.414678 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.414654 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b4c98b6b-w7lch" event={"ID":"0dae54b3-dceb-4795-85ec-42f9fb9ff256","Type":"ContainerStarted","Data":"c4ab2cfa8ceb7546c2058faf373f9b324a7aafcaf5d65b102adbdd9bd438e86f"} Oct 31 00:19:22.417196 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.417166 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qg2xf" event={"ID":"abe218cc-f275-4474-a54e-3ba513e969c2","Type":"ContainerStarted","Data":"efc548d3a48b897e56147e39dafebc31cef2494e7a81a7cf88e47fea6aeaaff7"} Oct 31 00:19:22.433896 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.433865 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf"] Oct 31 00:19:22.434988 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:22.434960 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a38f09d_12b6_481c_bb13_c686226854a6.slice/crio-233b42da5a4c25707657e89ae5f607c5dcf36e4b7582f5b14642b8f29cc8dddc WatchSource:0}: Error finding container 233b42da5a4c25707657e89ae5f607c5dcf36e4b7582f5b14642b8f29cc8dddc: Status 404 returned error can't find the container with id 233b42da5a4c25707657e89ae5f607c5dcf36e4b7582f5b14642b8f29cc8dddc Oct 31 00:19:22.457262 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.457218 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kqx7w" podStartSLOduration=1.437336749 podStartE2EDuration="4.457204458s" totalImagesPullingTime="3.019867709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:22.456728881 +0000 UTC m=+50.936905451" watchObservedRunningTime="2026-10-31 00:19:22.457204458 +0000 UTC m=+50.937381030" Oct 31 00:19:22.500874 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.500797 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf"] Oct 31 00:19:22.509518 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:22.509161 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qg2xf" podStartSLOduration=1.934999272 podStartE2EDuration="4.509140707s" totalImagesPullingTime="2.574141435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:19:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:22.508154425 +0000 UTC m=+50.988330997" watchObservedRunningTime="2026-10-31 00:19:22.509140707 +0000 UTC m=+50.989317281" Oct 31 00:19:23.018277 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.018238 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:23.289216 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.287437 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:19:23.290661 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.290632 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.302651 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.302549 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:23.311150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.310328 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:19:23.311150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.310566 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:19:23.311150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.310630 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:19:23.311150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.310684 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:19:23.311150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.310806 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vhm6s\"" Oct 31 00:19:23.311150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.310886 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:19:23.311150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.310982 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:19:23.311150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.311048 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:19:23.311150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.311106 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:19:23.324197 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.324163 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:19:23.334336 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.334302 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:19:23.426951 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.426910 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" event={"ID":"8a38f09d-12b6-481c-bb13-c686226854a6","Type":"ContainerStarted","Data":"c334f8f22c50fc6557089f7b6e222b7e5c55e6fe202ae16152eceececd9aa3c8"} Oct 31 00:19:23.427410 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.426990 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" event={"ID":"8a38f09d-12b6-481c-bb13-c686226854a6","Type":"ContainerStarted","Data":"d51a9350470a2d4bb5706ec01a753a8f691f7b25a3ee31e144588b418ccb3c80"} Oct 31 00:19:23.427410 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.427024 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" event={"ID":"8a38f09d-12b6-481c-bb13-c686226854a6","Type":"ContainerStarted","Data":"233b42da5a4c25707657e89ae5f607c5dcf36e4b7582f5b14642b8f29cc8dddc"} Oct 31 00:19:23.429240 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.429211 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcdc4b64f-p6xjm" event={"ID":"0fa8cd46-8095-4e93-b23a-0a422a47eab3","Type":"ContainerStarted","Data":"e975cdcf919ecf22befb787d8d84d9f8d259de811c42246a389919bcac9e5531"} Oct 31 00:19:23.432171 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.431753 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" event={"ID":"364a8716-433d-4181-94ed-f6ccd33a0828","Type":"ContainerStarted","Data":"8435fab2a04c233afd15453c1b699fa8f07822a04a5161bed1d97aa01f29491a"} Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452648 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb89\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452695 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452724 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452828 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452865 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452891 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452921 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452961 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.452988 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.453025 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.453056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.453354 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.453086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.505403 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.505352 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5dcdc4b64f-p6xjm" podStartSLOduration=1.912093049 podStartE2EDuration="5.505335879s" totalImagesPullingTime="3.59324283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:23.503098809 +0000 UTC m=+51.983275382" watchObservedRunningTime="2026-10-31 00:19:23.505335879 +0000 UTC m=+51.985512452" Oct 31 00:19:23.554333 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.554251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.554912 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.554991 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555019 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555090 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb89\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555139 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555332 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555383 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555415 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.555796 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.556820 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.556210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.570977 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.570009 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.570977 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.570235 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.571707 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.571648 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.572446 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.572421 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.572446 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.572431 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.572940 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.572916 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.573013 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.572996 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.573013 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.573004 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.574044 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.574019 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.595478 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.595428 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb89\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.596265 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.596235 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:23.614138 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.613895 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:23.968834 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:23.968804 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:19:23.974704 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:23.974673 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ede307_5582_4953_abaa_25d4393dc55c.slice/crio-685c759650d3bc6423a5850f50eee487f651bd04752375115fc0ce094a86a517 WatchSource:0}: Error finding container 685c759650d3bc6423a5850f50eee487f651bd04752375115fc0ce094a86a517: Status 404 returned error can't find the container with id 685c759650d3bc6423a5850f50eee487f651bd04752375115fc0ce094a86a517 Oct 31 00:19:24.350007 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.349967 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr"] Oct 31 00:19:24.352491 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.352466 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.364776 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.364745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.364776 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.364778 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.365019 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.364804 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.365019 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.364910 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.365019 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.364944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb59038-b544-4cb1-a2af-ded4bb5d6431-metrics-client-ca\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.365104 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.365051 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.365137 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.365111 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cb59038-b544-4cb1-a2af-ded4bb5d6431-tmp\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.365197 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.365175 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-grpc-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.365231 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.365208 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hphs\" (UniqueName: \"kubernetes.io/projected/2cb59038-b544-4cb1-a2af-ded4bb5d6431-kube-api-access-2hphs\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.386002 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.385978 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr"] Oct 31 00:19:24.407539 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.407320 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:19:24.407539 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.407391 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:19:24.407539 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.407480 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:19:24.407539 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.407500 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9vtvc\"" Oct 31 00:19:24.407844 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.407830 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e3j1gt3835o86\"" Oct 31 00:19:24.408092 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.408078 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:19:24.409163 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.409147 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:19:24.437974 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.437931 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" event={"ID":"364a8716-433d-4181-94ed-f6ccd33a0828","Type":"ContainerStarted","Data":"702e8a31f873073d840ee6c6f69677d5c580afab1ed2707c8416d909a33397f2"} Oct 31 00:19:24.438347 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.438009 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" event={"ID":"364a8716-433d-4181-94ed-f6ccd33a0828","Type":"ContainerStarted","Data":"825a2a77c0fe5954dea839c0b83ac64265ba48f78d5bf080a6d7d36f632ad0a9"} Oct 31 00:19:24.438347 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.438041 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" event={"ID":"364a8716-433d-4181-94ed-f6ccd33a0828","Type":"ContainerStarted","Data":"27db31cb2ffe6a8312c6b38955670a7875701a5c942f3be1f6447c03a0d86bd1"} Oct 31 00:19:24.439800 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.439775 2572 generic.go:356] "Generic (PLEG): container finished" podID="465a3984-adbc-4677-b22f-9e9240a770f8" containerID="8dca9a5de6099dcdfb08a7b240b416b5ac61810caedc53dd45eae1a140871bbb" exitCode=0 Oct 31 00:19:24.439918 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.439861 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lhrff" event={"ID":"465a3984-adbc-4677-b22f-9e9240a770f8","Type":"ContainerDied","Data":"8dca9a5de6099dcdfb08a7b240b416b5ac61810caedc53dd45eae1a140871bbb"} Oct 31 00:19:24.443065 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.443042 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" event={"ID":"8a38f09d-12b6-481c-bb13-c686226854a6","Type":"ContainerStarted","Data":"eed82769478b9d54272605a8c42f2cf64076169a9fd2b4046161c6e3c6950c86"} Oct 31 00:19:24.444825 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.444745 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerStarted","Data":"685c759650d3bc6423a5850f50eee487f651bd04752375115fc0ce094a86a517"} Oct 31 00:19:24.466904 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.466869 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cb59038-b544-4cb1-a2af-ded4bb5d6431-tmp\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.467040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.466949 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-grpc-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.467040 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.466976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hphs\" (UniqueName: \"kubernetes.io/projected/2cb59038-b544-4cb1-a2af-ded4bb5d6431-kube-api-access-2hphs\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.467124 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.467054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.467124 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.467083 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.467124 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.467113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.467219 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.467180 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.467342 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.467314 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb59038-b544-4cb1-a2af-ded4bb5d6431-metrics-client-ca\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.467444 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.467428 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.468735 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.468699 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cb59038-b544-4cb1-a2af-ded4bb5d6431-tmp\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.472935 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.472649 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb59038-b544-4cb1-a2af-ded4bb5d6431-metrics-client-ca\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.474225 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.474169 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.474875 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.474849 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.475117 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.475043 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-grpc-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.475250 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.475225 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.475360 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.475345 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.476766 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.476742 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.502648 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.502595 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hphs\" (UniqueName: \"kubernetes.io/projected/2cb59038-b544-4cb1-a2af-ded4bb5d6431-kube-api-access-2hphs\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.552485 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.551324 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" podStartSLOduration=2.427993794 podStartE2EDuration="3.55131057s" totalImagesPullingTime="1.123316776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:24.550645106 +0000 UTC m=+53.030821692" watchObservedRunningTime="2026-10-31 00:19:24.55131057 +0000 UTC m=+53.031487142" Oct 31 00:19:24.552485 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.552306 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" podStartSLOduration=2.149693284 podStartE2EDuration="3.552295997s" totalImagesPullingTime="1.402602713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:24.498270207 +0000 UTC m=+52.978446801" watchObservedRunningTime="2026-10-31 00:19:24.552295997 +0000 UTC m=+53.032472571" Oct 31 00:19:24.568416 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.568370 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:24.670343 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.670254 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:24.988123 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:24.988100 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr"] Oct 31 00:19:24.996195 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:24.996164 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb59038_b544_4cb1_a2af_ded4bb5d6431.slice/crio-7a926bc16dea92103ccd863b22095c76a7c6e1cdb9bab8d77f0047173febbed4 WatchSource:0}: Error finding container 7a926bc16dea92103ccd863b22095c76a7c6e1cdb9bab8d77f0047173febbed4: Status 404 returned error can't find the container with id 7a926bc16dea92103ccd863b22095c76a7c6e1cdb9bab8d77f0047173febbed4 Oct 31 00:19:25.451769 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:25.451725 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" event={"ID":"2cb59038-b544-4cb1-a2af-ded4bb5d6431","Type":"ContainerStarted","Data":"7a926bc16dea92103ccd863b22095c76a7c6e1cdb9bab8d77f0047173febbed4"} Oct 31 00:19:25.456836 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:25.456799 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lhrff" event={"ID":"465a3984-adbc-4677-b22f-9e9240a770f8","Type":"ContainerStarted","Data":"3c15beeef3530574db68eb54b6310601ee25625515bbe5bb1930e58b08545c3d"} Oct 31 00:19:25.456980 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:25.456869 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lhrff" event={"ID":"465a3984-adbc-4677-b22f-9e9240a770f8","Type":"ContainerStarted","Data":"d548631467b05a48c4f570d0a9a005e8bc54bf630c946eb346cf4096e3e692f2"} Oct 31 00:19:25.459322 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:25.459252 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b4c98b6b-w7lch" event={"ID":"0dae54b3-dceb-4795-85ec-42f9fb9ff256","Type":"ContainerStarted","Data":"7dba283abe3a374371a9fe2b35905e6b1cca9b421835f0bddb66506b02d20dbc"} Oct 31 00:19:25.461352 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:25.461326 2572 generic.go:356] "Generic (PLEG): container finished" podID="90ede307-5582-4953-abaa-25d4393dc55c" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" exitCode=0 Oct 31 00:19:25.461514 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:25.461405 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerDied","Data":"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd"} Oct 31 00:19:25.508425 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:25.508371 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lhrff" podStartSLOduration=3.254110966 podStartE2EDuration="4.508355589s" totalImagesPullingTime="1.171871102s" totalInitContainerRuntime="82.373521ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:25.508269606 +0000 UTC m=+53.988446182" watchObservedRunningTime="2026-10-31 00:19:25.508355589 +0000 UTC m=+53.988532165" Oct 31 00:19:25.564740 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:25.564687 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69b4c98b6b-w7lch" podStartSLOduration=2.548726349 podStartE2EDuration="4.564671672s" totalImagesPullingTime="2.015945323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:25.564165548 +0000 UTC m=+54.044342158" watchObservedRunningTime="2026-10-31 00:19:25.564671672 +0000 UTC m=+54.044848245" Oct 31 00:19:26.125923 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.125886 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:26.383375 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.383343 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6cbf58948-7xm9f"] Oct 31 00:19:26.385583 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.385557 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.406532 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.406507 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:19:26.406688 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.406642 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:19:26.419438 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.419404 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cbf58948-7xm9f"] Oct 31 00:19:26.437793 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.437747 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:19:26.438059 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.438034 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jhm6q\"" Oct 31 00:19:26.438163 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.438108 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:19:26.445585 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.445526 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9h6ltk5ck35h9\"" Oct 31 00:19:26.486853 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.486821 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-metrics-server-audit-profiles\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.487320 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.486885 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.487320 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.487215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-tls\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.487320 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.487286 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4c5s\" (UniqueName: \"kubernetes.io/projected/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-kube-api-access-s4c5s\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.487700 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.487679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-client-certs\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.487802 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.487784 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-audit-log\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.487862 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.487843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-client-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.539929 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.539890 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-br9bn"] Oct 31 00:19:26.542526 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.542495 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:26.562155 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.562118 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4z9hp\"" Oct 31 00:19:26.569294 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.568978 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:19:26.574043 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.574015 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-br9bn"] Oct 31 00:19:26.588324 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.588284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-tls\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.588512 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.588338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c5s\" (UniqueName: \"kubernetes.io/projected/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-kube-api-access-s4c5s\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.588512 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.588393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-client-certs\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.588512 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.588439 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-audit-log\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.588512 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.588502 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-client-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.588725 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.588540 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-metrics-server-audit-profiles\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.588725 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.588578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.588725 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.588700 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719455f-b7a1-4394-8258-6d84f4c0ccd3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:26.588862 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.588726 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr978\" (UniqueName: \"kubernetes.io/projected/a719455f-b7a1-4394-8258-6d84f4c0ccd3-kube-api-access-mr978\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:26.590424 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.589160 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-audit-log\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.590424 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.589831 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-metrics-server-audit-profiles\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.590424 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.590192 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.592170 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.592141 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-client-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.593815 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.593794 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-tls\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.604169 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.604141 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-client-certs\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.614357 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.614316 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4c5s\" (UniqueName: \"kubernetes.io/projected/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-kube-api-access-s4c5s\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.619811 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.619777 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:26.689769 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.689680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719455f-b7a1-4394-8258-6d84f4c0ccd3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:26.689769 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.689724 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr978\" (UniqueName: \"kubernetes.io/projected/a719455f-b7a1-4394-8258-6d84f4c0ccd3-kube-api-access-mr978\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:26.692667 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.692609 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719455f-b7a1-4394-8258-6d84f4c0ccd3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:26.697335 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.697303 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:26.722042 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.722009 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr978\" (UniqueName: \"kubernetes.io/projected/a719455f-b7a1-4394-8258-6d84f4c0ccd3-kube-api-access-mr978\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:26.815445 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.815354 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:26.856137 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:26.856094 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:27.019368 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:27.018525 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cbf58948-7xm9f"] Oct 31 00:19:27.024512 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:27.024473 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0bd8aa_5627_4ece_8ef4_97a529ab89ae.slice/crio-5597b293105e772e2058ab64663d3591692b4cbb16a2db98d8574e9c025e682e WatchSource:0}: Error finding container 5597b293105e772e2058ab64663d3591692b4cbb16a2db98d8574e9c025e682e: Status 404 returned error can't find the container with id 5597b293105e772e2058ab64663d3591692b4cbb16a2db98d8574e9c025e682e Oct 31 00:19:27.178307 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:27.178283 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-br9bn"] Oct 31 00:19:27.180879 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:27.180842 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda719455f_b7a1_4394_8258_6d84f4c0ccd3.slice/crio-53e4475911c055d82cf3d7a32e091e3206839855d51308278b9e877475378553 WatchSource:0}: Error finding container 53e4475911c055d82cf3d7a32e091e3206839855d51308278b9e877475378553: Status 404 returned error can't find the container with id 53e4475911c055d82cf3d7a32e091e3206839855d51308278b9e877475378553 Oct 31 00:19:27.473548 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:27.473504 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" event={"ID":"2cb59038-b544-4cb1-a2af-ded4bb5d6431","Type":"ContainerStarted","Data":"b686e92797dc0d65895b8a31d3a9d5a3466e39a520885289f3d7f6f8b0c21deb"} Oct 31 00:19:27.473692 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:27.473601 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" event={"ID":"2cb59038-b544-4cb1-a2af-ded4bb5d6431","Type":"ContainerStarted","Data":"d04089d37fc5b094da3e2fc7f2ad93827378146cb9ee48bf0f245093e630c9cf"} Oct 31 00:19:27.473692 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:27.473642 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" event={"ID":"2cb59038-b544-4cb1-a2af-ded4bb5d6431","Type":"ContainerStarted","Data":"5ecda0f939fba820ea03ef96241f1b603e29f96b1094d7c4c88df2ff8425bdfb"} Oct 31 00:19:27.475140 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:27.475111 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" event={"ID":"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae","Type":"ContainerStarted","Data":"5597b293105e772e2058ab64663d3591692b4cbb16a2db98d8574e9c025e682e"} Oct 31 00:19:27.476497 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:27.476431 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" event={"ID":"a719455f-b7a1-4394-8258-6d84f4c0ccd3","Type":"ContainerStarted","Data":"53e4475911c055d82cf3d7a32e091e3206839855d51308278b9e877475378553"} Oct 31 00:19:27.479858 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:27.479755 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerStarted","Data":"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7"} Oct 31 00:19:27.479858 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:27.479812 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerStarted","Data":"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7"} Oct 31 00:19:28.488931 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.488891 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerStarted","Data":"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad"} Oct 31 00:19:28.489465 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.488975 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerStarted","Data":"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7"} Oct 31 00:19:28.489465 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.489005 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerStarted","Data":"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45"} Oct 31 00:19:28.489465 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.489045 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerStarted","Data":"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8"} Oct 31 00:19:28.492638 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.492600 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" event={"ID":"2cb59038-b544-4cb1-a2af-ded4bb5d6431","Type":"ContainerStarted","Data":"4bbc26f69e56c985d1c11dcca21d55b5e3f1a6d71f1cf285542c2b2d5c1ff8b9"} Oct 31 00:19:28.492781 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.492695 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" event={"ID":"2cb59038-b544-4cb1-a2af-ded4bb5d6431","Type":"ContainerStarted","Data":"f6e819b58f7d5f6a8d974cc149ecc87aa53b27c1cdd26e882c2efcd212f914d3"} Oct 31 00:19:28.492781 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.492736 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" event={"ID":"2cb59038-b544-4cb1-a2af-ded4bb5d6431","Type":"ContainerStarted","Data":"910afcbab2f55ade09ea8aba07804333939b56e6c80c7e9e29d4a10706eefdfe"} Oct 31 00:19:28.492888 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.492794 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:28.642291 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.641212 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" podStartSLOduration=1.71334627 podStartE2EDuration="4.641189378s" totalImagesPullingTime="2.927843108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:28.640078538 +0000 UTC m=+57.120255112" watchObservedRunningTime="2026-10-31 00:19:28.641189378 +0000 UTC m=+57.121365952" Oct 31 00:19:28.642291 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.641543 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.060157891 podStartE2EDuration="5.641533548s" totalImagesPullingTime="3.568848029s" totalInitContainerRuntime="12.527628ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:28.577726315 +0000 UTC m=+57.057902924" watchObservedRunningTime="2026-10-31 00:19:28.641533548 +0000 UTC m=+57.121710122" Oct 31 00:19:28.923022 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.922938 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:28.923022 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.923025 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:28.927969 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:28.927941 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:29.076652 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.076617 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:29.342956 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.342916 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:29.353325 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.353296 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:19:29.356940 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.356917 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.384682 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.384655 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:19:29.385094 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.385075 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:19:29.385262 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.385247 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:19:29.406344 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.406316 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:19:29.412135 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.412105 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:19:29.420430 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420400 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.420603 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420436 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.420603 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420476 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.420603 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420579 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.420754 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420619 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.420754 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420665 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.420754 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420685 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.420754 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420716 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:19:29.420907 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420774 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.420907 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420794 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.420907 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420879 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421002 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420919 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421002 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420940 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421002 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421002 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.420998 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.421071 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421150 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.421096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwthq\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421245 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.421161 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421245 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.421209 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421307 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.421288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.421338 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.421323 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.450472 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.450410 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:19:29.450645 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.450627 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:19:29.450711 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.450655 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:19:29.450766 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.450709 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:19:29.450833 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.450820 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cno706g3ng8gr\"" Oct 31 00:19:29.450865 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.450829 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:19:29.450943 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.450930 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:19:29.450991 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.450979 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:19:29.451029 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.450983 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:19:29.451078 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.451027 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:19:29.451132 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.451122 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mg4tg\"" Oct 31 00:19:29.498137 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.498099 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" event={"ID":"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae","Type":"ContainerStarted","Data":"d08e5e6ac8ad3e0add8cb023b856bbdc0af2f40b6c96d11adc42cd7d3e77d19c"} Oct 31 00:19:29.499744 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.499717 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" event={"ID":"a719455f-b7a1-4394-8258-6d84f4c0ccd3","Type":"ContainerStarted","Data":"dec813ef2bed3af17429115eed8ae8f6b2e54ef4ee43152f51d080a3f6da752b"} Oct 31 00:19:29.500002 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.499983 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:29.503848 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.503826 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:29.507862 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.506559 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:29.522686 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.522650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.522686 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.522686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwthq\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.522928 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.522726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.522928 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.522860 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523066 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523043 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523135 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523115 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523619 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523210 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523619 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523278 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523619 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523511 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523619 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523619 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523950 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523666 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523950 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523718 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523950 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523817 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523950 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.523950 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.523930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.524225 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.524061 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.524225 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.524088 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.524225 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.524113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.524225 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.524175 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.524225 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.524184 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.524501 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.524298 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.526121 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.525793 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.526121 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.526099 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.526507 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.526473 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.527134 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.527104 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.527519 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.527493 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.527636 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.527615 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.528439 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.528064 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.528439 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.528188 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.528439 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.528379 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.528731 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.528702 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.528731 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.528717 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.528857 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.528829 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.529147 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.529123 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.530036 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.530011 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.530195 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.530165 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.530436 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.530418 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.530975 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.530958 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.549142 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.549092 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" podStartSLOduration=2.057621673 podStartE2EDuration="3.549077257s" totalImagesPullingTime="1.491455584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:29.546892556 +0000 UTC m=+58.027069141" watchObservedRunningTime="2026-10-31 00:19:29.549077257 +0000 UTC m=+58.029253830" Oct 31 00:19:29.554788 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.554760 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwthq\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:29.566710 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.566675 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:29.660587 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.660467 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" podStartSLOduration=2.108113297 podStartE2EDuration="3.660434492s" totalImagesPullingTime="1.552321195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:29.658157353 +0000 UTC m=+58.138333949" watchObservedRunningTime="2026-10-31 00:19:29.660434492 +0000 UTC m=+58.140611065" Oct 31 00:19:29.682349 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:29.682318 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:30.003835 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:30.003809 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:19:30.005422 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:30.005396 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ec7561_c654_4156_9ec3_5cb5893881b8.slice/crio-5a446832b1b0a1c34d8f267e9f78e5eef1e4b8b3e25f0ce756fbf78403ed4593 WatchSource:0}: Error finding container 5a446832b1b0a1c34d8f267e9f78e5eef1e4b8b3e25f0ce756fbf78403ed4593: Status 404 returned error can't find the container with id 5a446832b1b0a1c34d8f267e9f78e5eef1e4b8b3e25f0ce756fbf78403ed4593 Oct 31 00:19:30.505130 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:30.505094 2572 generic.go:356] "Generic (PLEG): container finished" podID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerID="f94e4d0b68df9df554ee8f6a4eec3a65d85d4d43ac8df05dd72a3b0c381191cc" exitCode=0 Oct 31 00:19:30.505645 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:30.505194 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerDied","Data":"f94e4d0b68df9df554ee8f6a4eec3a65d85d4d43ac8df05dd72a3b0c381191cc"} Oct 31 00:19:30.505645 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:30.505265 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerStarted","Data":"5a446832b1b0a1c34d8f267e9f78e5eef1e4b8b3e25f0ce756fbf78403ed4593"} Oct 31 00:19:31.285113 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:31.285026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:31.420106 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:31.420078 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:31.485120 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:31.485061 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:31.564381 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:31.564306 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:32.063389 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:32.063360 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:32.063573 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:32.063418 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:32.065986 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:32.065962 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:32.518795 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:32.518762 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:32.598970 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:32.598934 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dcdc4b64f-p6xjm"] Oct 31 00:19:32.820295 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:32.820261 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:32.977238 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:32.977186 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:32.999317 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:32.999284 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-fb49695f-qnzhd"] Oct 31 00:19:33.003356 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.003328 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.029398 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.029362 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb49695f-qnzhd"] Oct 31 00:19:33.064010 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.063973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.064210 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.064040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.064210 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.064132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.064210 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.064204 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.064353 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.064248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.064353 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.064333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.064416 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.064359 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhtf\" (UniqueName: \"kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.064416 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.064399 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.165537 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.165425 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.165717 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.165578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.165717 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.165606 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.165717 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.165654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.165717 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.165695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.165923 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.165734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.165923 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.165757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhtf\" (UniqueName: \"kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.165923 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.165816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.166079 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.166016 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.166278 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.166256 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.167076 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.167048 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.167169 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.167088 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.167657 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.167634 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.168877 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.168851 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.169058 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.169035 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.193048 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.193016 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhtf\" (UniqueName: \"kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.318171 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.317581 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:33.624378 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:33.624342 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890a1f30_b435_43fd_9f63_a1123bd6e661.slice/crio-738d74e83e8dfae45a6748aabc87ca0bbcc8e5a3fd622d27c4a27e6907ede597 WatchSource:0}: Error finding container 738d74e83e8dfae45a6748aabc87ca0bbcc8e5a3fd622d27c4a27e6907ede597: Status 404 returned error can't find the container with id 738d74e83e8dfae45a6748aabc87ca0bbcc8e5a3fd622d27c4a27e6907ede597 Oct 31 00:19:33.624800 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:33.624775 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb49695f-qnzhd"] Oct 31 00:19:34.365339 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.365301 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:34.506818 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.506791 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:34.526509 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.526437 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerStarted","Data":"ac271606873784c21badc01029a2e1a3bb847c9ad7ab3bdf28725a4a083c3010"} Oct 31 00:19:34.526676 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.526541 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerStarted","Data":"1bf03022442047c5405a49f4b0aa49d12fcf1189aff9b1151a9890ee13d5b599"} Oct 31 00:19:34.526676 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.526585 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerStarted","Data":"01c06835981e0b68637cfca8ee4f2521cb0f12ecf90b85e80368126a98e67299"} Oct 31 00:19:34.526676 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.526620 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerStarted","Data":"6b2f498915bfa2be8ed136ce18d689919db669e615ceceb13314838421a5a9bb"} Oct 31 00:19:34.526676 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.526655 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerStarted","Data":"8c441888f2d6f0588f264a3c462f47388be96dcb0127c6c7858dce4078962ded"} Oct 31 00:19:34.526899 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.526693 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerStarted","Data":"6aba1a03018f85efb484fdd0764a442f278064adbce382d60d4e23c945be49fe"} Oct 31 00:19:34.528064 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.528043 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb49695f-qnzhd" event={"ID":"890a1f30-b435-43fd-9f63-a1123bd6e661","Type":"ContainerStarted","Data":"d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6"} Oct 31 00:19:34.528184 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.528081 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb49695f-qnzhd" event={"ID":"890a1f30-b435-43fd-9f63-a1123bd6e661","Type":"ContainerStarted","Data":"738d74e83e8dfae45a6748aabc87ca0bbcc8e5a3fd622d27c4a27e6907ede597"} Oct 31 00:19:34.531648 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.531619 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:34.628388 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.628272 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-fb49695f-qnzhd" podStartSLOduration=2.628253237 podStartE2EDuration="2.628253237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:34.626842905 +0000 UTC m=+63.107019478" watchObservedRunningTime="2026-10-31 00:19:34.628253237 +0000 UTC m=+63.108429809" Oct 31 00:19:34.683163 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.683119 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:34.703269 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:34.703213 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.557987395 podStartE2EDuration="5.703198345s" totalImagesPullingTime="3.128427053s" totalInitContainerRuntime="16.783897ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:34.701871015 +0000 UTC m=+63.182047619" watchObservedRunningTime="2026-10-31 00:19:34.703198345 +0000 UTC m=+63.183374917" Oct 31 00:19:36.803292 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.803250 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:36.803734 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.803300 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:36.818299 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.818273 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:19:36.826783 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.826753 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:36.831557 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.831537 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:19:36.835800 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.835770 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:36.897864 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.897834 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dsrzd\"" Oct 31 00:19:36.903953 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.903925 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:36.904298 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.904271 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:36.919205 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.919182 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:19:36.939945 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.939911 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9wq7f\"" Oct 31 00:19:36.939945 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.939932 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:19:36.945055 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.945027 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:36.947641 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:36.947614 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:37.203751 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:37.203704 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56c984b_c9d9_4d3f_86fd_2314f01a6673.slice/crio-ef40440a75872ddb6694256ad651088eae4af4cc3ffe3a0ad2d2957656d2f61c WatchSource:0}: Error finding container ef40440a75872ddb6694256ad651088eae4af4cc3ffe3a0ad2d2957656d2f61c: Status 404 returned error can't find the container with id ef40440a75872ddb6694256ad651088eae4af4cc3ffe3a0ad2d2957656d2f61c Oct 31 00:19:37.203889 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:37.203794 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z4hfm"] Oct 31 00:19:37.219736 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:37.219711 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6b92m\"" Oct 31 00:19:37.224773 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:37.224733 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:37.253504 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:37.253475 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xzjcg"] Oct 31 00:19:37.253870 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:37.253844 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57df0df3_a039_45eb_8ae2_02433f6b0b2e.slice/crio-959d3dfaab1975953dd5c1f81aecfd5be3a5caf6a57ddb662e766940731e3a99 WatchSource:0}: Error finding container 959d3dfaab1975953dd5c1f81aecfd5be3a5caf6a57ddb662e766940731e3a99: Status 404 returned error can't find the container with id 959d3dfaab1975953dd5c1f81aecfd5be3a5caf6a57ddb662e766940731e3a99 Oct 31 00:19:37.522986 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:37.522956 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mswcp"] Oct 31 00:19:37.523127 ip-10-0-129-199 kubenswrapper[2572]: W1031 00:19:37.523070 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6249a4c3_cd70_4c06_aa16_9864e6493086.slice/crio-b888ba970d0931c387eb2e7e175f032569826053368610de4d95852dfc93956a WatchSource:0}: Error finding container b888ba970d0931c387eb2e7e175f032569826053368610de4d95852dfc93956a: Status 404 returned error can't find the container with id b888ba970d0931c387eb2e7e175f032569826053368610de4d95852dfc93956a Oct 31 00:19:37.547134 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:37.547099 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mswcp" event={"ID":"6249a4c3-cd70-4c06-aa16-9864e6493086","Type":"ContainerStarted","Data":"b888ba970d0931c387eb2e7e175f032569826053368610de4d95852dfc93956a"} Oct 31 00:19:37.548307 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:37.548280 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z4hfm" event={"ID":"a56c984b-c9d9-4d3f-86fd-2314f01a6673","Type":"ContainerStarted","Data":"ef40440a75872ddb6694256ad651088eae4af4cc3ffe3a0ad2d2957656d2f61c"} Oct 31 00:19:37.549327 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:37.549306 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzjcg" event={"ID":"57df0df3-a039-45eb-8ae2-02433f6b0b2e","Type":"ContainerStarted","Data":"959d3dfaab1975953dd5c1f81aecfd5be3a5caf6a57ddb662e766940731e3a99"} Oct 31 00:19:38.557875 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:38.557782 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzjcg" event={"ID":"57df0df3-a039-45eb-8ae2-02433f6b0b2e","Type":"ContainerStarted","Data":"f04bc17cd2c73ebd2f97975ede45d8746a93f230ac61dbc0ca83002622f00323"} Oct 31 00:19:38.557875 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:38.557848 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzjcg" event={"ID":"57df0df3-a039-45eb-8ae2-02433f6b0b2e","Type":"ContainerStarted","Data":"de728210d7324efb473432ece9bf25c8ee1a3299bd6a292046e1fd650546d2b8"} Oct 31 00:19:38.607109 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:38.606490 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xzjcg" podStartSLOduration=65.67060711 podStartE2EDuration="1m6.606446863s" totalImagesPullingTime="935.839753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:38.60449422 +0000 UTC m=+67.084670795" watchObservedRunningTime="2026-10-31 00:19:38.606446863 +0000 UTC m=+67.086623437" Oct 31 00:19:38.902768 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:38.902729 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:39.193429 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:39.193356 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:40.478737 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:40.478701 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:40.666558 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:40.666514 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:41.274241 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:41.274200 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:41.531599 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:41.531513 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:41.574733 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:41.574695 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mswcp" event={"ID":"6249a4c3-cd70-4c06-aa16-9864e6493086","Type":"ContainerStarted","Data":"abf0464b03d63e09719e071b7d4c24cd5a3eeea9724b701923a4dc9af1d17453"} Oct 31 00:19:41.574901 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:41.574830 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:41.621971 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:41.621907 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mswcp" podStartSLOduration=66.503440825 podStartE2EDuration="1m9.621893212s" totalImagesPullingTime="3.118452387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:41.620937768 +0000 UTC m=+70.101114342" watchObservedRunningTime="2026-10-31 00:19:41.621893212 +0000 UTC m=+70.102069784" Oct 31 00:19:42.221970 ip-10-0-129-199 kubenswrapper[2572]: I1031 00:19:42.221477 2572 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:19:42.221620 ip-10-0-129-199 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:19:42.235875 ip-10-0-129-199 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:19:42.236055 ip-10-0-129-199 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:19:42.236113 ip-10-0-129-199 systemd[1]: kubelet.service: Consumed 5.199s CPU time. Oct 31 00:19:42.255296 ip-10-0-129-199 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:19:42.334153 ip-10-0-129-199 kubenswrapper[5978]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:19:42.334153 ip-10-0-129-199 kubenswrapper[5978]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:19:42.334153 ip-10-0-129-199 kubenswrapper[5978]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:19:42.334153 ip-10-0-129-199 kubenswrapper[5978]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:19:42.336299 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336210 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:19:42.336299 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336298 5978 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:19:42.336299 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336304 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336307 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336310 5978 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336314 5978 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336317 5978 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336320 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336328 5978 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336331 5978 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336334 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336339 5978 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336344 5978 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336347 5978 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336350 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336352 5978 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336355 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336358 5978 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336361 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336364 5978 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336366 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:19:42.336399 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336369 5978 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336371 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336374 5978 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336377 5978 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336379 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336382 5978 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336384 5978 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336388 5978 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336390 5978 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336394 5978 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336397 5978 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336399 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336402 5978 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336405 5978 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336408 5978 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336411 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336413 5978 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336417 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336419 5978 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336422 5978 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:19:42.336888 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336425 5978 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336427 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336430 5978 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336432 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336435 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336437 5978 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336440 5978 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336442 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336445 5978 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336448 5978 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336464 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336467 5978 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336469 5978 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336472 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336474 5978 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336477 5978 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336480 5978 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336483 5978 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336486 5978 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:19:42.337418 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336488 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336491 5978 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336494 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336497 5978 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336500 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336502 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336507 5978 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336510 5978 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336513 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336516 5978 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336519 5978 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336521 5978 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336525 5978 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336527 5978 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336530 5978 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336533 5978 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336536 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336538 5978 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336541 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:19:42.337886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336543 5978 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336546 5978 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336548 5978 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336551 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336553 5978 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336556 5978 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336558 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336560 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336563 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336565 5978 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336568 5978 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336571 5978 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336573 5978 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336576 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336578 5978 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336582 5978 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336585 5978 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336587 5978 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336590 5978 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336592 5978 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:19:42.338369 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336595 5978 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336597 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336601 5978 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336603 5978 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336606 5978 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336610 5978 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.336613 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337419 5978 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337426 5978 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337429 5978 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337432 5978 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337435 5978 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337438 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337441 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337444 5978 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337447 5978 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337463 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337466 5978 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337469 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337472 5978 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:19:42.338871 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337475 5978 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337477 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337481 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337484 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337487 5978 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337490 5978 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337492 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337496 5978 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337499 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337502 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337504 5978 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337507 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337510 5978 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337513 5978 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337516 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337518 5978 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337521 5978 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337523 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337526 5978 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337532 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:19:42.339378 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337535 5978 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337538 5978 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337540 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337543 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337545 5978 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337548 5978 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337551 5978 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337554 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337557 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337559 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337564 5978 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337567 5978 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337570 5978 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337573 5978 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337575 5978 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337579 5978 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337587 5978 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337590 5978 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337593 5978 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:19:42.339882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337597 5978 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337602 5978 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337605 5978 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337608 5978 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337611 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337614 5978 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337616 5978 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337619 5978 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337621 5978 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337624 5978 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337626 5978 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337629 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337631 5978 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337639 5978 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337642 5978 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337645 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337647 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337650 5978 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337653 5978 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337655 5978 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:19:42.340349 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337658 5978 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337660 5978 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337663 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337666 5978 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337668 5978 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337671 5978 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337674 5978 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337677 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337679 5978 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337682 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337684 5978 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337687 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337689 5978 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337692 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337694 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337697 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337700 5978 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337702 5978 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337704 5978 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:19:42.340864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337707 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337709 5978 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337713 5978 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337715 5978 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337718 5978 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337720 5978 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337723 5978 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337731 5978 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337734 5978 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337736 5978 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337739 5978 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337742 5978 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337744 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337747 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.337751 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338266 5978 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338271 5978 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338275 5978 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338279 5978 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338282 5978 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:19:42.341323 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338284 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338287 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338290 5978 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338292 5978 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338295 5978 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338297 5978 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338300 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338303 5978 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338305 5978 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338307 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338310 5978 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338312 5978 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338316 5978 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338318 5978 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338321 5978 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338323 5978 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338326 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338329 5978 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338332 5978 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:19:42.341815 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338335 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338337 5978 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338340 5978 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338342 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338345 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338348 5978 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338350 5978 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338353 5978 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338355 5978 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338358 5978 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338361 5978 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338363 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338365 5978 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338368 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338371 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338373 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338376 5978 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338379 5978 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338381 5978 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:19:42.342308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338384 5978 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338386 5978 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338389 5978 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338392 5978 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338394 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338396 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338399 5978 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338401 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338404 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338406 5978 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338411 5978 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338414 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338418 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338424 5978 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338428 5978 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338431 5978 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338433 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338436 5978 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338439 5978 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:19:42.342779 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338441 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338444 5978 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338447 5978 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338462 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338465 5978 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338468 5978 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338470 5978 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338473 5978 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338476 5978 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338478 5978 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338481 5978 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338490 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338495 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338498 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338503 5978 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338506 5978 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338509 5978 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338512 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338515 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:19:42.343233 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338517 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338520 5978 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338523 5978 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338525 5978 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338528 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338531 5978 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338534 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338540 5978 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338543 5978 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338546 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338549 5978 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338552 5978 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338555 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338557 5978 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338560 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338563 5978 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338565 5978 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338568 5978 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338570 5978 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338573 5978 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:19:42.343717 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338576 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338578 5978 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338581 5978 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338583 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.338587 5978 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338682 5978 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338697 5978 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338704 5978 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338711 5978 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338716 5978 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338719 5978 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338724 5978 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338728 5978 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338731 5978 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338735 5978 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338738 5978 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338742 5978 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338745 5978 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338748 5978 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338752 5978 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338756 5978 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338758 5978 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338763 5978 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338768 5978 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:19:42.344184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338771 5978 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338774 5978 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338778 5978 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338781 5978 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338785 5978 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338788 5978 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338791 5978 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338795 5978 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338798 5978 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338801 5978 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338804 5978 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338808 5978 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338811 5978 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338815 5978 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338818 5978 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338821 5978 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338826 5978 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338830 5978 flags.go:64] FLAG: --enable-server="true" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338833 5978 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338838 5978 flags.go:64] FLAG: --event-burst="100" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338841 5978 flags.go:64] FLAG: --event-qps="50" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338844 5978 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338847 5978 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338850 5978 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:19:42.344965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338854 5978 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338857 5978 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338860 5978 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338865 5978 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338868 5978 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338871 5978 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338875 5978 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338879 5978 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338882 5978 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338885 5978 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338888 5978 flags.go:64] FLAG: --feature-gates="" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338892 5978 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338895 5978 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338898 5978 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338901 5978 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338904 5978 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338908 5978 flags.go:64] FLAG: --help="false" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338911 5978 flags.go:64] FLAG: --hostname-override="ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338914 5978 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338917 5978 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338920 5978 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338924 5978 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338927 5978 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338930 5978 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:19:42.345882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338935 5978 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338938 5978 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338941 5978 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338945 5978 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338948 5978 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338951 5978 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338954 5978 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338956 5978 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338960 5978 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338963 5978 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338966 5978 flags.go:64] FLAG: --lock-file="" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338970 5978 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338973 5978 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338976 5978 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338982 5978 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338984 5978 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338988 5978 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338991 5978 flags.go:64] FLAG: --logging-format="text" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338994 5978 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.338997 5978 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339000 5978 flags.go:64] FLAG: --manifest-url="" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339003 5978 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339008 5978 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339011 5978 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339015 5978 flags.go:64] FLAG: --max-pods="110" Oct 31 00:19:42.346514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339018 5978 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339021 5978 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339024 5978 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339027 5978 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339030 5978 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339033 5978 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339036 5978 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339044 5978 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339048 5978 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339051 5978 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339054 5978 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339057 5978 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339060 5978 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339063 5978 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339066 5978 flags.go:64] FLAG: --port="10250" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339070 5978 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339072 5978 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03c16c5961cd995d9" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339076 5978 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339080 5978 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339083 5978 flags.go:64] FLAG: --register-node="true" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339086 5978 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339090 5978 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339093 5978 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339096 5978 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339099 5978 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339104 5978 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:19:42.347112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339107 5978 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339110 5978 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339113 5978 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339116 5978 flags.go:64] FLAG: --runonce="false" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339119 5978 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339121 5978 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339124 5978 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339127 5978 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339130 5978 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339133 5978 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339137 5978 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339139 5978 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339142 5978 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339147 5978 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339151 5978 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339154 5978 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339157 5978 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339160 5978 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339163 5978 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339168 5978 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339171 5978 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339174 5978 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339179 5978 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339181 5978 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339185 5978 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:19:42.347804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339189 5978 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339192 5978 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339195 5978 flags.go:64] FLAG: --v="2" Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339199 5978 flags.go:64] FLAG: --version="false" Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339203 5978 flags.go:64] FLAG: --vmodule="" Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339207 5978 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339211 5978 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339364 5978 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339368 5978 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339372 5978 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339375 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339378 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339381 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339384 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339386 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339389 5978 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339393 5978 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339398 5978 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339401 5978 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339404 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339408 5978 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:19:42.348643 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339411 5978 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339414 5978 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339416 5978 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339419 5978 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339422 5978 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339424 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339427 5978 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339429 5978 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339432 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339435 5978 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339439 5978 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339441 5978 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339444 5978 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339446 5978 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339461 5978 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339464 5978 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339467 5978 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339469 5978 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339473 5978 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339476 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:19:42.349160 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339479 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339482 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339484 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339487 5978 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339490 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339493 5978 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339496 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339498 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339500 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339503 5978 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339506 5978 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339511 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339514 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339516 5978 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339519 5978 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339522 5978 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339525 5978 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339527 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339530 5978 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:19:42.349670 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339533 5978 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339535 5978 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339538 5978 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339542 5978 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339544 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339547 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339550 5978 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339552 5978 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339555 5978 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339558 5978 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339560 5978 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339563 5978 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339567 5978 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339570 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339572 5978 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339575 5978 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339577 5978 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339580 5978 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339583 5978 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339585 5978 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:19:42.350302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339588 5978 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339591 5978 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339593 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339596 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339601 5978 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339605 5978 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339608 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339611 5978 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339614 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339617 5978 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339619 5978 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339622 5978 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339624 5978 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339627 5978 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339630 5978 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339633 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339636 5978 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339639 5978 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339642 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:19:42.350864 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339644 5978 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339647 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339649 5978 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339652 5978 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339655 5978 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339657 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339661 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339663 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339666 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339668 5978 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339671 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339674 5978 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339676 5978 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339679 5978 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.339706 5978 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339741 5978 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:19:42.351409 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339745 5978 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339748 5978 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339752 5978 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339755 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339758 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339761 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339764 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339766 5978 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339769 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339772 5978 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339774 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339777 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339782 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339784 5978 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339787 5978 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339789 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339792 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339795 5978 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339797 5978 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:19:42.351909 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339800 5978 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339803 5978 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339805 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339808 5978 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339811 5978 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339814 5978 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339817 5978 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339819 5978 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339822 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339824 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339827 5978 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339829 5978 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339832 5978 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339834 5978 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339837 5978 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339841 5978 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339844 5978 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339846 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339849 5978 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:19:42.352373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339852 5978 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339855 5978 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339857 5978 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339860 5978 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339862 5978 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339865 5978 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339869 5978 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339872 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339875 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339878 5978 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339883 5978 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339886 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339889 5978 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339893 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339896 5978 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339899 5978 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339902 5978 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339905 5978 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339908 5978 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:19:42.352878 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339910 5978 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339913 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339916 5978 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339918 5978 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339921 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339923 5978 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339926 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339929 5978 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339933 5978 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339936 5978 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339939 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339942 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339944 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339947 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339949 5978 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339952 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339955 5978 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339957 5978 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339960 5978 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339963 5978 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:19:42.353442 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339966 5978 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339968 5978 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339971 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339973 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339976 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339979 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339981 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339984 5978 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339988 5978 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339991 5978 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339994 5978 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.339997 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340001 5978 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340003 5978 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340006 5978 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340009 5978 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340012 5978 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340015 5978 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340017 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:19:42.354164 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340019 5978 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340022 5978 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340025 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340028 5978 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340030 5978 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340033 5978 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340036 5978 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340038 5978 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.340041 5978 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.344311 5978 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.344331 5978 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.344349 5978 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.344354 5978 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344800 5978 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344874 5978 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344880 5978 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344892 5978 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344896 5978 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344901 5978 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344905 5978 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:19:42.354772 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344910 5978 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344914 5978 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344919 5978 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344923 5978 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344927 5978 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344931 5978 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344936 5978 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344940 5978 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344949 5978 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344953 5978 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344957 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344961 5978 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344965 5978 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344970 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344974 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344979 5978 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344984 5978 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344988 5978 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344993 5978 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.344997 5978 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:19:42.355259 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345001 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345010 5978 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345014 5978 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345018 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345023 5978 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345028 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345058 5978 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345063 5978 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345067 5978 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345071 5978 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345075 5978 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345079 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345083 5978 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345088 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345096 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345101 5978 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345106 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345110 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345115 5978 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345119 5978 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:19:42.355793 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345123 5978 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345128 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345132 5978 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345136 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345140 5978 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345145 5978 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345155 5978 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345160 5978 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345164 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345169 5978 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345174 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345178 5978 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345182 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345186 5978 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345217 5978 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345223 5978 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345228 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345233 5978 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345237 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:19:42.356294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345270 5978 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345275 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345280 5978 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345284 5978 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345290 5978 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345296 5978 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345301 5978 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345305 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345309 5978 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345314 5978 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345318 5978 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345322 5978 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345330 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345335 5978 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345339 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345343 5978 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345347 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345351 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345355 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:19:42.356859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345360 5978 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345364 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345368 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345373 5978 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345377 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345381 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345390 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345394 5978 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345399 5978 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345403 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345408 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345413 5978 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345417 5978 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345421 5978 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345507 5978 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345513 5978 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345517 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345521 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345526 5978 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:19:42.357373 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345535 5978 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.345539 5978 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.345607 5978 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346108 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346116 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346120 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346123 5978 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346126 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346128 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346131 5978 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346134 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346137 5978 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346139 5978 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346142 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346145 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346148 5978 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:19:42.357853 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346150 5978 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346154 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346157 5978 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346159 5978 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346162 5978 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346165 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346168 5978 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346171 5978 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346173 5978 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346176 5978 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346179 5978 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346181 5978 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346185 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346187 5978 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346190 5978 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346193 5978 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346195 5978 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346198 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346200 5978 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346203 5978 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:19:42.358288 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346206 5978 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346209 5978 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346211 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346214 5978 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346216 5978 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346219 5978 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346222 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346224 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346227 5978 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346229 5978 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346232 5978 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346235 5978 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346238 5978 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346241 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346243 5978 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346246 5978 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346248 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346251 5978 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346254 5978 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346257 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:19:42.358823 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346259 5978 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346262 5978 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346264 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346267 5978 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346270 5978 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346272 5978 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346276 5978 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346281 5978 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346285 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346287 5978 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346290 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346293 5978 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346295 5978 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346298 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346300 5978 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346303 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346305 5978 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346308 5978 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346310 5978 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:19:42.359312 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346313 5978 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346316 5978 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346318 5978 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346321 5978 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346324 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346327 5978 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346330 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346332 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346335 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346337 5978 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346340 5978 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346342 5978 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346345 5978 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346348 5978 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346350 5978 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346353 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346356 5978 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346360 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346364 5978 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:19:42.359773 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346367 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346370 5978 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346373 5978 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346376 5978 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346378 5978 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346381 5978 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346384 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346386 5978 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346389 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346391 5978 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346394 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346400 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346403 5978 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346406 5978 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346408 5978 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.346437 5978 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:19:42.360286 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346504 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346512 5978 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346517 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346520 5978 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346525 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346528 5978 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346531 5978 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346534 5978 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346536 5978 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346539 5978 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346542 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346544 5978 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346547 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346550 5978 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346553 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346555 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346559 5978 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346562 5978 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346564 5978 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:19:42.360761 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346567 5978 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346570 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346572 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346575 5978 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346578 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346580 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346583 5978 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346586 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346589 5978 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346591 5978 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346594 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346596 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346599 5978 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346601 5978 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346604 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346607 5978 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346610 5978 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346612 5978 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346615 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:19:42.361443 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346618 5978 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346620 5978 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346623 5978 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346625 5978 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346628 5978 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346631 5978 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346633 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346636 5978 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346639 5978 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346641 5978 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346644 5978 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346646 5978 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346649 5978 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346652 5978 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346655 5978 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346658 5978 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346660 5978 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346663 5978 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346665 5978 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346668 5978 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:19:42.362240 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346670 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346673 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346676 5978 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346678 5978 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346681 5978 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346683 5978 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346686 5978 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346688 5978 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346691 5978 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346695 5978 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346697 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346700 5978 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346702 5978 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346705 5978 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346708 5978 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346710 5978 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346713 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346716 5978 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346718 5978 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346721 5978 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:19:42.362875 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346725 5978 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346728 5978 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346731 5978 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346734 5978 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346736 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346739 5978 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346741 5978 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346744 5978 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346747 5978 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346749 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346752 5978 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346754 5978 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346757 5978 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346760 5978 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346763 5978 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346765 5978 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346768 5978 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346770 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346773 5978 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:19:42.363420 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346775 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346778 5978 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346780 5978 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346783 5978 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346786 5978 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346789 5978 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346791 5978 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346797 5978 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:42.346800 5978 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.346874 5978 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.347935 5978 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.347995 5978 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.348175 5978 server.go:1014] "Starting client certificate rotation" Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.348280 5978 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.348355 5978 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:13:32 +0000 UTC" deadline="2028-04-18 02:34:13.959781305 +0000 UTC" Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.348397 5978 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12842h14m31.611388321s" Oct 31 00:19:42.364065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.348659 5978 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:19:42.364716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.349842 5978 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:19:42.364716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.351187 5978 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:19:42.364716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.353165 5978 log.go:25] "Validated CRI v1 image API" Oct 31 00:19:42.364716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.353923 5978 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:19:42.364716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.356369 5978 fs.go:135] Filesystem UUIDs: map[470fd304-ba8e-4cc8-b4d7-d6565a532618:/dev/nvme0n1p4 62f55614-cd56-434a-a03c-273f37a7fe4d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 00:19:42.364947 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.356392 5978 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14346ce38b021388565ff9c29e76dd9908641220894197e95fe63cd377c093d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14346ce38b021388565ff9c29e76dd9908641220894197e95fe63cd377c093d8/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/233b42da5a4c25707657e89ae5f607c5dcf36e4b7582f5b14642b8f29cc8dddc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/233b42da5a4c25707657e89ae5f607c5dcf36e4b7582f5b14642b8f29cc8dddc/userdata/shm major:0 minor:269 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32d47e0e1ce59bc6c4787f4b8e8028b6abf07054417ca8d6c0d4838b95d05581/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32d47e0e1ce59bc6c4787f4b8e8028b6abf07054417ca8d6c0d4838b95d05581/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53e4475911c055d82cf3d7a32e091e3206839855d51308278b9e877475378553/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53e4475911c055d82cf3d7a32e091e3206839855d51308278b9e877475378553/userdata/shm major:0 minor:372 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5597b293105e772e2058ab64663d3591692b4cbb16a2db98d8574e9c025e682e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5597b293105e772e2058ab64663d3591692b4cbb16a2db98d8574e9c025e682e/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a446832b1b0a1c34d8f267e9f78e5eef1e4b8b3e25f0ce756fbf78403ed4593/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a446832b1b0a1c34d8f267e9f78e5eef1e4b8b3e25f0ce756fbf78403ed4593/userdata/shm major:0 minor:464 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f002c7dbb510dd14f973c08630a802a79cc6ae52e621603bd4d9d78d16292eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f002c7dbb510dd14f973c08630a802a79cc6ae52e621603bd4d9d78d16292eb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/606f1de1f43b9934f13b8fe518dffe1d71692f202bbbc2d49e76ef0e988e8bbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/606f1de1f43b9934f13b8fe518dffe1d71692f202bbbc2d49e76ef0e988e8bbf/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62560de8d575f7f73af9da33245d44cee439e0f9890f3e71b795e6595c2c475d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62560de8d575f7f73af9da33245d44cee439e0f9890f3e71b795e6595c2c475d/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/685c759650d3bc6423a5850f50eee487f651bd04752375115fc0ce094a86a517/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/685c759650d3bc6423a5850f50eee487f651bd04752375115fc0ce094a86a517/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c40f3a146031134a5ef8f7edad95480466c4028e385fabc50b348ddf6742a54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c40f3a146031134a5ef8f7edad95480466c4028e385fabc50b348ddf6742a54/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/738d74e83e8dfae45a6748aabc87ca0bbcc8e5a3fd622d27c4a27e6907ede597/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/738d74e83e8dfae45a6748aabc87ca0bbcc8e5a3fd622d27c4a27e6907ede597/userdata/shm major:0 minor:471 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a926bc16dea92103ccd863b22095c76a7c6e1cdb9bab8d77f0047173febbed4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a926bc16dea92103ccd863b22095c76a7c6e1cdb9bab8d77f0047173febbed4/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8435fab2a04c233afd15453c1b699fa8f07822a04a5161bed1d97aa01f29491a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8435fab2a04c233afd15453c1b699fa8f07822a04a5161bed1d97aa01f29491a/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88b0dbdd3eeacc6d7a65b3c38121d08f26c12b62eb27287c394adbbf1af68fc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88b0dbdd3eeacc6d7a65b3c38121d08f26c12b62eb27287c394adbbf1af68fc5/userdata/shm major:0 minor:275 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89377edc5335c112e15939e07830a6b41884d0d911ca703e93dc72dd7fd502cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89377edc5335c112e15939e07830a6b41884d0d911ca703e93dc72dd7fd502cc/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/959d3dfaab1975953dd5c1f81aecfd5be3a5caf6a57ddb662e766940731e3a99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/959d3dfaab1975953dd5c1f81aecfd5be3a5caf6a57ddb662e766940731e3a99/userdata/shm major:0 minor:517 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9841ac2eaecc67e3dea7f7ac1b0e211b0822c3ae099e21343a3a36f62b636ab6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9841ac2eaecc67e3dea7f7ac1b0e211b0822c3ae099e21343a3a36f62b636ab6/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b888ba970d0931c387eb2e7e175f032569826053368610de4d95852dfc93956a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b888ba970d0931c387eb2e7e175f032569826053368610de4d95852dfc93956a/userdata/shm major:0 minor:521 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c24bf0edadced92dbed653eb1521f7d88b866b801b4dbfd222bd9314aaf51849/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c24bf0edadced92dbed653eb1521f7d88b866b801b4dbfd222bd9314aaf51849/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4ab2cfa8ceb7546c2058faf373f9b324a7aafcaf5d65b102adbdd9bd438e86f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4ab2cfa8ceb7546c2058faf373f9b324a7aafcaf5d65b102adbdd9bd438e86f/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dac696112bcd1e84da4742935af48bb50d04e30f60e0d9a417e721be6ef95fb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dac696112bcd1e84da4742935af48bb50d04e30f60e0d9a417e721be6ef95fb3/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6691222484d14b0c92a573776082782a7b8cfb0674b0154c2ae1a3089d8123e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6691222484d14b0c92a573776082782a7b8cfb0674b0154c2ae1a3089d8123e/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eab2e57560f114f1accf98cdb298edb1a4f619f6f09723003a23cd02a81f0a33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eab2e57560f114f1accf98cdb298edb1a4f619f6f09723003a23cd02a81f0a33/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef40440a75872ddb6694256ad651088eae4af4cc3ffe3a0ad2d2957656d2f61c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef40440a75872ddb6694256ad651088eae4af4cc3ffe3a0ad2d2957656d2f61c/userdata/shm major:0 minor:514 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f95f9e90756a204a140abb4e56732e931cda0f25c18a8d2ba3d504610a3c90df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f95f9e90756a204a140abb4e56732e931cda0f25c18a8d2ba3d504610a3c90df/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/feae4182e7d2d731c263c8b3ddfa1e81e707fef6583ba1cb18aa78ed7682c00f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/feae4182e7d2d731c263c8b3ddfa1e81e707fef6583ba1cb18aa78ed7682c00f/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05e75b3b-ffb1-4cda-9203-f4882bd385e7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/05e75b3b-ffb1-4cda-9203-f4882bd385e7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05e75b3b-ffb1-4cda-9203-f4882bd385e7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/05e75b3b-ffb1-4cda-9203-f4882bd385e7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05e75b3b-ffb1-4cda-9203-f4882bd385e7/volumes/kubernetes.io~projected/kube-api-access-tfzq2:{mountpoint:/var/lib/kubelet/pods/05e75b3b-ffb1-4cda-9203-f4882bd385e7/volumes/kubernetes.io~projected/kube-api-access-tfzq2 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/064705a9-54fd-4528-ae88-1cb4596c5b6e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/064705a9-54fd-4528-ae88-1cb4596c5b6e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/064705a9-54fd-4528-ae88-1cb4596c5b6e/volumes/kubernetes.io~projected/kube-api-access-j8rbb:{mountpoint:/var/lib/kubelet/pods/064705a9-54fd-4528-ae88-1cb4596c5b6e/volumes/kubernetes.io~projected/kube-api-access-j8rbb major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/064705a9-54fd-4528-ae88-1cb4596c5b6e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/064705a9-54fd-4528-ae88-1cb4596c5b6e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes/kubernetes.io~projected/kube-api-access-tzj5s:{mountpoint:/var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes/kubernetes.io~projected/kube-api-access-tzj5s major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~projected/kube-api-access-s4c5s:{mountpoint:/var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~projected/kube-api-access-s4c5s major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes/kubernetes.io~projected/kube-api-access-pnvmn:{mountpoint:/var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes/kubernetes.io~projected/kube-api-access-pnvmn major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volume-subpaths/web-config/config-reloader/2 major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volume-subpaths/web-config/prometheus/15 major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~empty-dir/config-out major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~projected/kube-api-access-kwthq:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~projected/kube-api-access-kwthq major:0 minor:463 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~projected/tls-assets major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/config major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/web-config major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f9f082d-9176-491a-8d5e-281ee6665dae/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1f9f082d-9176-491a-8d5e-281ee6665dae/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b1af480-ae8f-483e-bd52-947094fcc3ff/volumes/kubernetes.io~projected/kube-api-access-jfq6z:{mountpoint:/var/lib/kubelet/pods/2b1af480-ae8f-483e-bd52-947094fcc3ff/volumes/kubernetes.io~projected/kube-api-access-jfq6z major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~projected/kube-api-access-2hphs:{mountpoint:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~projected/kube-api-access-2hphs major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d60169a-68c9-4cfd-bbd7-73ecac659695/volumes/kubernetes.io~projected/kube-api-access-2v5nl:{mountpoint:/var/lib/kubelet/pods/2d60169a-68c9-4cfd-bbd7-73ecac659695/volumes/kubernetes.io~projected/kube-api-access-2v5nl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/364a8716-433d-4181-94ed-f6ccd33a0828/volumes/kubernetes.io~projected/kube-api-access-6m2pd:{mountpoint:/var/lib/kubelet/pods/364a8716-433d-4181-94ed-f6ccd33a0828/volumes/kubernetes.io~projected/kube-api-access-6m2pd major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/364a8716-433d-4181-94ed-f6ccd33a0828/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/364a8716-433d-4181-94ed-f6ccd33a0828/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/364a8716-433d-4181-94ed-f6ccd33a0828/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/364a8716-433d-4181-94ed-f6ccd33a0828/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3af6b8f1-b029-4ae2-99cb-df0315daa268/volumes/kubernetes.io~projected/kube-api-access-5dlwf:{mountpoint:/var/lib/kubelet/pods/3af6b8f1-b029-4ae2-99cb-df0315daa268/volumes/kubernetes.io~projected/kube-api-access-5dlwf major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/465a3984-adbc-4677-b22f-9e9240a770f8/volumes/kubernetes.io~projected/kube-api-access-rwv2b:{mountpoint:/var/lib/kubelet/pods/465a3984-adbc-4677-b22f-9e9240a770f8/volumes/kubernetes.io~projected/kube-api-access-rwv2b major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/465a3984-adbc-4677-b22f-9e9240a770f8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/465a3984-adbc-4677-b22f-9e9240a770f8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/465a3984-adbc-4677-b22f-9e9240a770f8/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/465a3984-adbc-4677-b22f-9e9240a770f8/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47350ce3-ba5e-4393-ac0f-653b41bc28fd/volumes/kubernetes.io~projected/kube-api-access-flpkx:{mountpoint:/var/lib/kubelet/pods/47350ce3-ba5e-4393-ac0f-653b41bc28fd/volumes/kubernetes.io~projected/kube-api-access-flpkx major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57df0df3-a039-45eb-8ae2-02433f6b0b2e/volumes/kubernetes.io~projected/kube-api-access-8x95q:{mountpoint:/var/lib/kubelet/pods/57df0df3-a039-45eb-8ae2-02433f6b0b2e/volumes/kubernetes.io~projected/kube-api-access-8x95q major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57df0df3-a039-45eb-8ae2-02433f6b0b2e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/57df0df3-a039-45eb-8ae2-02433f6b0b2e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:513 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6249a4c3-cd70-4c06-aa16-9864e6493086/volumes/kubernetes.io~projected/kube-api-access-chgxt:{mountpoint:/var/lib/kubelet/pods/6249a4c3-cd70-4c06-aa16-9864e6493086/volumes/kubernetes.io~projected/kube-api-access-chgxt major:0 minor:516 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dd2bec9-6796-4e26-ad19-bfd15842852b/volumes/kubernetes.io~projected/kube-api-access-nkrbb:{mountpoint:/var/lib/kubelet/pods/7dd2bec9-6796-4e26-ad19-bfd15842852b/volumes/kubernetes.io~projected/kube-api-access-nkrbb major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dd2bec9-6796-4e26-ad19-bfd15842852b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7dd2bec9-6796-4e26-ad19-bfd15842852b/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes/kubernetes.io~projected/kube-api-access-9hhtf:{mountpoint:/var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes/kubernetes.io~projected/kube-api-access-9hhtf major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:468 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a38f09d-12b6-481c-bb13-c686226854a6/volumes/kubernetes.io~projected/kube-api-access-drh6b:{mountpoint:/var/lib/kubelet/pods/8a38f09d-12b6-481c-bb13-c686226854a6/volumes/kubernetes.io~projected/kube-api-access-drh6b major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a38f09d-12b6-481c-bb13-c686226854a6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8a38f09d-12b6-481c-bb13-c686226854a6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a38f09d-12b6-481c-bb13-c686226854a6/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/8a38f09d-12b6-481c-bb13-c686226854a6/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/web-config/alertmanager/9 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/web-config/config-reloader/6 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~projected/kube-api-access-8mb89:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~projected/kube-api-access-8mb89 major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~projected/tls-assets major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/config-volume major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/web-config major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a56c984b-c9d9-4d3f-86fd-2314f01a6673/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a56c984b-c9d9-4d3f-86fd-2314f01a6673/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:509 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a719455f-b7a1-4394-8258-6d84f4c0ccd3/volumes/kubernetes.io~projected/kube-api-access-mr978:{mountpoint:/var/lib/kubelet/pods/a719455f-b7a1-4394-8258-6d84f4c0ccd3/volumes/kubernetes.io~projected/kube-api-access-mr978 major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a719455f-b7a1-4394-8258-6d84f4c0ccd3/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/a719455f-b7a1-4394-8258-6d84f4c0ccd3/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abe218cc-f275-4474-a54e-3ba513e969c2/volumes/kubernetes.io~projected/kube-api-access-khv2q:{mountpoint:/var/lib/kubelet/pods/abe218cc-f275-4474-a54e-3ba513e969c2/volumes/kubernetes.io~projected/kube-api-access-khv2q major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abe218cc-f275-4474-a54e-3ba513e969c2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/abe218cc-f275-4474-a54e-3ba513e969c2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b70404c4-e809-42c9-9efc-d6262e52815c/volumes/kubernetes.io~projected/kube-api-access-2sk42:{mountpoint:/var/lib/kubelet/pods/b70404c4-e809-42c9-9efc-d6262e52815c/volumes/kubernetes.io~projected/kube-api-access-2sk42 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6f8e29e-fb86-4153-b168-0bb1743e44cd/volumes/kubernetes.io~projected/kube-api-access-7kl9k:{mountpoint:/var/lib/kubelet/pods/d6f8e29e-fb86-4153-b168-0bb1743e44cd/volumes/kubernetes.io~projected/kube-api-access-7kl9k major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6f8e29e-fb86-4153-b168-0bb1743e44cd/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d6f8e29e-fb86-4153-b168-0bb1743e44cd/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f99f8918-3e3d-456b-be91-db4120b67424/volumes/kubernetes.io~projected/kube-api-access-76qh9:{mountpoint:/var/lib/kubelet/pods/f99f8918-3e3d-456b-be91-db4120b67424/volumes/kubernetes.io~projected/kube-api-access-76qh9 major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/38d6a5524d97ef4aab7c91c9a4c9222426c09793a43fab90002f41ea12b2b8d1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2cb991f32c064d1b1405919742ef2341ff1ddd22d977ddc29e82616023590858/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a0d995ccf8af61f1f02c415248383cdb73039f0a370092d24dc89bd341600668/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/cb48c37c287c2e94aa97b700a54e5622084f3a7a9888fd5a0d5854747396ccdf/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/c272f96d8b185e6d6561f40a62fbe6729cbf0ed9095a8ea58234a8149e152d74/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/22b2df5d7999967f5475b37001b59e0204c4bcd5d22df7b86658f5eac41a59d4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/16416bef9886810f8b3b39e8fa4ddb73441085065876da2e018d1fa82b85c97d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d22781c3f20a13aef2a15cb43f3cc142bccf9a8eb048cdc578f44ad6465ed495/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/2a12d33f89e10100f4b03ceab7a4d6f2c0f6d36c022d747bfd07c9703631a4ee/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/5a85272da5f343666cda6b2915ba5f998ddbe655a3e1555f2249b9983c0f89bd/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/e9da0c4cc2cb4b9b6a1b4624af087891a3e1d0a2d4e7545ec3bea12772bfbd8a/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/9ec1822316562654669b29c02ba6bc180f458b4a90b6e941034e69633b6b825c/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c475bf4b17cb53f3dd88a87009eb49cdf49a07733e5053f49f14a125591ef5b9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ebbc70cd2718f274de911b3d37fd67800cc331f46622819021d0d69d21bb9271/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bff64e32f8ee75f303ce01ccc5b95aba3461e2be79ed5f050613cc474a00eb3b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f8c31dd6ae507ba73c82fc8bbcad2bd83003835954c913ba1017402fad9e14b8/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ea303221e9c946fe3978cdfc8cb69bbf77186a525688700367e875e9700b63d4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4eecccc1ae86f8d00a566851d7af8078f4db9f9d915bd9f4bcc623ae86685c0c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f768776cc9f1bafabac1d8cfb10db98912eab3e529a7ae6529b035feec2587be/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/65a2cfc08329bfdf07d86fc18406c3b486fdeec3be3794f91a108af743bd360e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1014df7f52e6044fa4f9ec2d77e088380683e3d35f2578690fe52316d22fa78e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/6efe1bb7e26116c331930586af11af47e41a01aeee31e7ef046233ac9cdd92c8/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/3dcd969a23f33f9bbd65c3565540669fdb7f75243ffd3b7a870a33fcdce0db4c/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/6b1e32a22706e56eea851f7bdc14e8d7a653b0b062441f887e5934c8b0907e3b/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/85e44b3aeb17a84ab86d53b849181440c581cceaf22777a0235360fb30e3b6e5/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/677701700be4711c074cd0b8ab2f94e649d138fd180be7656e6145cd5ed8a5e4/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/cf981f17fa80ea0143d173477a1883b4cf6615cba3fbd64e0e42970d4a1042c3/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/23db5890e68cd3c0db88aef1f1587c27347449cbf316ada99297b2bb4b762894/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/176d75c0736fa9cabc2a72988513e95f78445b43e116f8c7923ffdb8a9d19576/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/55830b5529e972d86341162c4b8a8e7a394c27f32986849be5067173e60b32b6/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/4da0faba2b9358b31f6516861f320c6edce57bda8659303d8bf3115957186abc/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/06bd5d15f5e5c9951dbb264458c5c2e616dcfea01566ea892c984fc9f9c264ff/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/b0d846c0b43793d77b9d54626b7d0a27a392b334dd18d59e7b8b1093f0565eca/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/f80cc09631cbd43e9f12026f9a08cd6fd2cede95c463f415c1f65460fe784824/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/16b45cb0e1722bd0c7299fdcf96568d79af1e64f47be510637ce1df9a8c11db6/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/aa610b09a8b575df728d8b8dcaa0bd81a29582bce72244b99c12c9f7b1eb94db/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/6d5a7cd6da8f273f11f68af2cbb8eddda067af669429b7b950737e6c92866965/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/08b08175f766617cfc098c09f6f46441a50b0f9424901f3860fdcd47b52ebf3a/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/2986f4deee1c1607749588a1a4a05337507ffab6501c324bda43a24dcd8a463f/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/3e9a875cd5ba259332898f991d914892c5a41081e3e0c9173ee48088bca6eecb/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/52598b8a7646a0c3cb26dc1cadd21f8a57f6e86af7a54559ee70ac199113bc1d/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/f3f9a3d11833af6ad5af8c94cd615e4a575e378ed0d14e57d7869a25ab326ead/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/f02e108ac949680f971ae1c20655250269f9e28c9544ffe910fb9e6dc30bc30a/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/149730b67bdba00d94d725bb155bfe5628da2a79f2bb05ec3d6ea5021e230737/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/8f3b83b60655ea07d049ab9d3df7455dabc9e0b211ffed8f05e6cc57619a7890/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/4ad0ed903190f1b153587931d6f6830e8cfa46373a4fa0bf44e7a6f2789a1449/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/4f1a7d91377b26b98a236d1bd7bd6a519c387627637ba1fda3e4788156128d19/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/5a55e13091d55bec4b3abf5a68e4a0787ddcc383d7f2f1e062054f72a5a342dc/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/acdf31f463e6cb3ba8a4b0093171ac9e0b9e5d172ec0bd613cae9cf8f789a436/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/025db756d7d03f6839da5dd1bce3e717cb8209eff1efad3ac5c3f639a9134e04/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/4ae42cda6645cd419f743e552535bb32d9d65970f6ef97a645ee27afb264de76/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/7caac3ad4a4502c73bd5d912723272be52b291cae650882b86ca6a47985b3598/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/13c505202d6af4f69343517fd0487aca6ec833ff728c7a045266c9719122d5e6/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/3c80bb7f9b26eee2c4873496ae9e203011853595e6391417efaf47c9c65a6308/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/ab0a19814cd5ff6634933aaa4479497978883da46aeaa527dcc9ebe9e3735c3a/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/f92f6edd8972249665e619e754ddcaee74c23fb95b5c7416168777f932ac9ba8/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/3fd36722b64de39d078ad18482deacd6204d86843562b478c0045bd751799fd8/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/b3b327c5c2d40a4233ccb3125da387702a9fd82c29bcd50a539aee89af7b408e/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/8e9dbf3c204101a37b12bda1b431ced8d807a98dcb99df4dd13678b82f52dcf8/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/ad5fac48a6a39ace27dc64c8628af34023d005ba4d6a34ddceb6be39bfc91172/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/9d4887223fc2a30085ade6609b9acc381c3fe5f3630e26fd71b50629c530f529/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/2ded3b7d3caa2cb882bbe414d4dddba83615364032dbd05729c01b24159b1afd/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/dd287017a01530852b8cd8609b67a7bff722cb2db44f3229446a651832e38225/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/5072f52b24f7896e8f749e3671acd9fcf6a081ddfe3ee864b82e37d80f463769/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/04b944bef6aa39b8ed93e6d3c5c9d495513f99a59b4f76616e8e023e47a7d9df/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/c17cc83d9c1bf1115140ba7f9d8278b97e7da69f2289260f86034e0183317d68/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/60398f1bd42efde90ec9bdfc84ef589ec915b526d035e3765296abc0851259aa/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-479:{mountpoint:/var/lib/containers/storage/overlay/3670c7b3a005c08c3384e2f5e45fd8a35656d2f7310792ed4d9d2ead7ea41e00/merged major:0 minor:479 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/ee719ae426737dcb7d48876ae1727dfb7a5864ce53e52044c7f502a0ff470cb3/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-494:{mountpoint:/var/lib/containers/storage/overlay/9c5b47500884cfb0fefc351b6ed9d2a33c657ef8623e40bb63e740f2203eddf3/merged major:0 minor:494 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/6aae013fa880317ccbd4105bdb8027712d525ec898f8a5fcead7e557e307bb0e/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-504:{mountpoint:/var/lib/containers/storage/overlay/7cb2593256afe87d7589a027ee4281dc219bf28860355950aa81ef74c05928ed/merged major:0 minor:504 fsType:overlay blockSize:0} overlay_0-519:{mountpoint:/var/lib/containers/storage/overlay/03392e5ff5e1c053096328fe2cc0bb352a27a7fc198dfa6005217e929514cfb5/merged major:0 minor:519 fsType:overlay blockSize:0} overlay_0-523:{mountpoint:/var/lib/containers/storage/overlay/ab5c9450dd3cd518c556a975cd403c8c76d8d951881991b67a99755e651275d6/merged major:0 minor:523 fsType:overlay blockSize:0} overlay_0-525:{mountpoint:/var/lib/containers/storage/overlay/dd58cd0bc618b46901e33d71d58b5fce287d2702b737636ac96cec125a29d2d6/merged major:0 minor:525 fsType:overlay blockSize:0} overlay_0-527:{mountpoint:/var/lib/containers/storage/overlay/9df3c933f4923dcab94e4ba1df5478faab73cf32723a09a0ffeca248fbb849ae/merged major:0 minor:527 fsType:overlay blockSize:0} overlay_0-529:{mountpoint:/var/lib/containers/storage/overlay/333d8a8d64a4a3aa3815610c94a4451d3607abc38f489f23ddc287c87e56619d/merged major:0 minor:529 fsType:overlay blockSize:0} overlay_0-535:{mountpoint:/var/lib/containers/storage/overlay/2aa97126a9374b8bab37f3921e97e1436b17cd7c9fb3ce37dba800a863037e2f/merged major:0 minor:535 fsType:overlay blockSize:0} overlay_0-540:{mountpoint:/var/lib/containers/storage/overlay/17c3199e246e5314415b7cad0bb160a7e7a00f609652802fe3f02dbcd23873db/merged major:0 minor:540 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/95d0fc5c0249e7d084f03ee2f51491655320b6ec22d6a498b68dfd45c6a99c65/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/4cc2e539a91ab5fcc46532d104d62bd203cf935b34e2e25c21bdb828ae361a0c/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/001fce61c0f5c4e47590d0bb2bc95e0fb63be37254366acacce4ad255beebc3d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6543defcfef1bcc8ac2108f52af24615bc16c19f2d74ee8ac1cc682b71a458cb/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9bca7bcc8469b4d3b391979362ac45616b67093a9cca82e1a9f96519e6e9bf82/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/57daef7e3c88808b67b2182fbf39e5b62cbe8b6d658cb7a5387392cc417a0649/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a6f62c8d4fca1bbe2631291b661eb13458360335408db5c75add8eec6dfd5430/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/97def8a3f077294f22c30bc398f285b926557cb17cc6789c88885bd148f11379/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/99a6a709acb3c75d2739bdba5d89ba245639abe9b1dee0f73df8e3b3add17c0a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e701bd6f62403f7f4806c2b0ecc2d6557a6b7a46d1d240320720247b9d9db2c2/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 00:19:42.366868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.366365 5978 manager.go:217] Machine: {Timestamp:2026-10-31 00:19:42.365883464 +0000 UTC m=+0.063997220 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3135954 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cdd1a4884d5506c6853617cbaed5a SystemUUID:ec2cdd1a-4884-d550-6c68-53617cbaed5a BootID:e1c92a72-893f-4d30-996a-0100f1d422d0 Filesystems:[{Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:457 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:469 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-504 DeviceMajor:0 DeviceMinor:504 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57df0df3-a039-45eb-8ae2-02433f6b0b2e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:513 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/959d3dfaab1975953dd5c1f81aecfd5be3a5caf6a57ddb662e766940731e3a99/userdata/shm DeviceMajor:0 DeviceMinor:517 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a38f09d-12b6-481c-bb13-c686226854a6/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:351 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/738d74e83e8dfae45a6748aabc87ca0bbcc8e5a3fd622d27c4a27e6907ede597/userdata/shm DeviceMajor:0 DeviceMinor:471 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05e75b3b-ffb1-4cda-9203-f4882bd385e7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/685c759650d3bc6423a5850f50eee487f651bd04752375115fc0ce094a86a517/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a926bc16dea92103ccd863b22095c76a7c6e1cdb9bab8d77f0047173febbed4/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a56c984b-c9d9-4d3f-86fd-2314f01a6673/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:509 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a38f09d-12b6-481c-bb13-c686226854a6/volumes/kubernetes.io~projected/kube-api-access-drh6b DeviceMajor:0 DeviceMinor:260 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a719455f-b7a1-4394-8258-6d84f4c0ccd3/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:368 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/47350ce3-ba5e-4393-ac0f-653b41bc28fd/volumes/kubernetes.io~projected/kube-api-access-flpkx DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89377edc5335c112e15939e07830a6b41884d0d911ca703e93dc72dd7fd502cc/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a446832b1b0a1c34d8f267e9f78e5eef1e4b8b3e25f0ce756fbf78403ed4593/userdata/shm DeviceMajor:0 DeviceMinor:464 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:468 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b888ba970d0931c387eb2e7e175f032569826053368610de4d95852dfc93956a/userdata/shm DeviceMajor:0 DeviceMinor:521 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/064705a9-54fd-4528-ae88-1cb4596c5b6e/volumes/kubernetes.io~projected/kube-api-access-j8rbb DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05e75b3b-ffb1-4cda-9203-f4882bd385e7/volumes/kubernetes.io~projected/kube-api-access-tfzq2 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62560de8d575f7f73af9da33245d44cee439e0f9890f3e71b795e6595c2c475d/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes/kubernetes.io~projected/kube-api-access-9hhtf DeviceMajor:0 DeviceMinor:470 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/064705a9-54fd-4528-ae88-1cb4596c5b6e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14346ce38b021388565ff9c29e76dd9908641220894197e95fe63cd377c093d8/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/feae4182e7d2d731c263c8b3ddfa1e81e707fef6583ba1cb18aa78ed7682c00f/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c40f3a146031134a5ef8f7edad95480466c4028e385fabc50b348ddf6742a54/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/465a3984-adbc-4677-b22f-9e9240a770f8/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:366 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abe218cc-f275-4474-a54e-3ba513e969c2/volumes/kubernetes.io~projected/kube-api-access-khv2q DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:455 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:364 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5597b293105e772e2058ab64663d3591692b4cbb16a2db98d8574e9c025e682e/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:455 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32d47e0e1ce59bc6c4787f4b8e8028b6abf07054417ca8d6c0d4838b95d05581/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/606f1de1f43b9934f13b8fe518dffe1d71692f202bbbc2d49e76ef0e988e8bbf/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/465a3984-adbc-4677-b22f-9e9240a770f8/volumes/kubernetes.io~projected/kube-api-access-rwv2b DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:462 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/57df0df3-a039-45eb-8ae2-02433f6b0b2e/volumes/kubernetes.io~projected/kube-api-access-8x95q DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abe218cc-f275-4474-a54e-3ba513e969c2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88b0dbdd3eeacc6d7a65b3c38121d08f26c12b62eb27287c394adbbf1af68fc5/userdata/shm DeviceMajor:0 DeviceMinor:275 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c24bf0edadced92dbed653eb1521f7d88b866b801b4dbfd222bd9314aaf51849/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:243 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:455 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d60169a-68c9-4cfd-bbd7-73ecac659695/volumes/kubernetes.io~projected/kube-api-access-2v5nl DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eab2e57560f114f1accf98cdb298edb1a4f619f6f09723003a23cd02a81f0a33/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~projected/kube-api-access-8mb89 DeviceMajor:0 DeviceMinor:311 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/1f9f082d-9176-491a-8d5e-281ee6665dae/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f002c7dbb510dd14f973c08630a802a79cc6ae52e621603bd4d9d78d16292eb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a719455f-b7a1-4394-8258-6d84f4c0ccd3/volumes/kubernetes.io~projected/kube-api-access-mr978 DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:454 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9841ac2eaecc67e3dea7f7ac1b0e211b0822c3ae099e21343a3a36f62b636ab6/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7dd2bec9-6796-4e26-ad19-bfd15842852b/volumes/kubernetes.io~projected/kube-api-access-nkrbb DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4ab2cfa8ceb7546c2058faf373f9b324a7aafcaf5d65b102adbdd9bd438e86f/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:461 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes/kubernetes.io~projected/kube-api-access-pnvmn DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef40440a75872ddb6694256ad651088eae4af4cc3ffe3a0ad2d2957656d2f61c/userdata/shm DeviceMajor:0 DeviceMinor:514 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7dd2bec9-6796-4e26-ad19-bfd15842852b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:456 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-519 DeviceMajor:0 DeviceMinor:519 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6691222484d14b0c92a573776082782a7b8cfb0674b0154c2ae1a3089d8123e/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/465a3984-adbc-4677-b22f-9e9240a770f8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:459 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:460 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:455 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dac696112bcd1e84da4742935af48bb50d04e30f60e0d9a417e721be6ef95fb3/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/364a8716-433d-4181-94ed-f6ccd33a0828/volumes/kubernetes.io~projected/kube-api-access-6m2pd DeviceMajor:0 DeviceMinor:262 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-494 DeviceMajor:0 DeviceMinor:494 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6249a4c3-cd70-4c06-aa16-9864e6493086/volumes/kubernetes.io~projected/kube-api-access-chgxt DeviceMajor:0 DeviceMinor:516 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-540 DeviceMajor:0 DeviceMinor:540 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:346 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:448 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-523 DeviceMajor:0 DeviceMinor:523 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-527 DeviceMajor:0 DeviceMinor:527 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b1af480-ae8f-483e-bd52-947094fcc3ff/volumes/kubernetes.io~projected/kube-api-access-jfq6z DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f99f8918-3e3d-456b-be91-db4120b67424/volumes/kubernetes.io~projected/kube-api-access-76qh9 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:247 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~projected/kube-api-access-2hphs DeviceMajor:0 DeviceMinor:355 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes/kubernetes.io~projected/kube-api-access-tzj5s DeviceMajor:0 DeviceMinor:248 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~projected/kube-api-access-s4c5s DeviceMajor:0 DeviceMinor:367 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-525 DeviceMajor:0 DeviceMinor:525 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-529 DeviceMajor:0 DeviceMinor:529 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6f8e29e-fb86-4153-b168-0bb1743e44cd/volumes/kubernetes.io~projected/kube-api-access-7kl9k DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/364a8716-433d-4181-94ed-f6ccd33a0828/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/364a8716-433d-4181-94ed-f6ccd33a0828/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-535 DeviceMajor:0 DeviceMinor:535 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6f8e29e-fb86-4153-b168-0bb1743e44cd/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/233b42da5a4c25707657e89ae5f607c5dcf36e4b7582f5b14642b8f29cc8dddc/userdata/shm DeviceMajor:0 DeviceMinor:269 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:365 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3af6b8f1-b029-4ae2-99cb-df0315daa268/volumes/kubernetes.io~projected/kube-api-access-5dlwf DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:458 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~projected/kube-api-access-kwthq DeviceMajor:0 DeviceMinor:463 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f95f9e90756a204a140abb4e56732e931cda0f25c18a8d2ba3d504610a3c90df/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8435fab2a04c233afd15453c1b699fa8f07822a04a5161bed1d97aa01f29491a/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:453 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-479 DeviceMajor:0 DeviceMinor:479 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/064705a9-54fd-4528-ae88-1cb4596c5b6e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/05e75b3b-ffb1-4cda-9203-f4882bd385e7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b70404c4-e809-42c9-9efc-d6262e52815c/volumes/kubernetes.io~projected/kube-api-access-2sk42 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53e4475911c055d82cf3d7a32e091e3206839855d51308278b9e877475378553/userdata/shm DeviceMajor:0 DeviceMinor:372 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a38f09d-12b6-481c-bb13-c686226854a6/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb59038-b544-4cb1-a2af-ded4bb5d6431/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:14346ce38b02138 MacAddress:f6:14:c8:9c:79:e1 Speed:10000 Mtu:8901} {Name:233b42da5a4c257 MacAddress:86:bd:f4:05:f3:43 Speed:10000 Mtu:8901} {Name:53e4475911c055d MacAddress:46:75:d5:60:95:c8 Speed:10000 Mtu:8901} {Name:5597b293105e772 MacAddress:8e:43:7e:2f:af:43 Speed:10000 Mtu:8901} {Name:5a446832b1b0a1c MacAddress:3e:84:46:6a:ac:b6 Speed:10000 Mtu:8901} {Name:606f1de1f43b993 MacAddress:1a:fc:6d:5e:e2:e8 Speed:10000 Mtu:8901} {Name:685c759650d3bc6 MacAddress:da:1e:7e:47:a2:a0 Speed:10000 Mtu:8901} {Name:738d74e83e8dfae MacAddress:8e:39:b6:1d:aa:e1 Speed:10000 Mtu:8901} {Name:7a926bc16dea921 MacAddress:a2:06:44:0c:29:cf Speed:10000 Mtu:8901} {Name:8435fab2a04c233 MacAddress:96:e6:c5:db:ec:53 Speed:10000 Mtu:8901} {Name:89377edc5335c11 MacAddress:1e:fd:4f:0b:fa:35 Speed:10000 Mtu:8901} {Name:959d3dfaab19759 MacAddress:62:60:79:e0:e5:88 Speed:10000 Mtu:8901} {Name:b888ba970d0931c MacAddress:ba:90:12:50:50:ba Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cc:6f:6a:57 Speed:0 Mtu:9001} {Name:br-int MacAddress:42:bb:c5:3b:b5:3f Speed:0 Mtu:8901} {Name:c4ab2cfa8ceb754 MacAddress:ee:28:d7:d7:87:2b Speed:10000 Mtu:8901} {Name:dac696112bcd1e8 MacAddress:fe:6f:55:c1:81:db Speed:10000 Mtu:8901} {Name:ef40440a75872dd MacAddress:da:ee:22:1f:a7:33 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cc:6f:6a:57 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:6e:65:06:64:7e:ca Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:e0:d1:0e:16:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:19:42.367149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.366875 5978 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:19:42.367149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.366927 5978 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:19:42.367245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367222 5978 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:19:42.367391 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367246 5978 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-199.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:19:42.367437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367394 5978 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:19:42.367437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367402 5978 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:19:42.367437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367413 5978 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:19:42.367437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367433 5978 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:19:42.367561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367553 5978 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:19:42.367725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367716 5978 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:19:42.367840 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367831 5978 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:19:42.367877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367868 5978 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:19:42.367909 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367882 5978 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:19:42.367909 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367893 5978 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:19:42.367909 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367902 5978 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:19:42.367909 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.367910 5978 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:19:42.368644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.368625 5978 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:19:42.368644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.368638 5978 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:19:42.369208 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.369187 5978 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:19:42.373019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.372999 5978 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:19:42.373019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373017 5978 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:19:42.373136 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373126 5978 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:19:42.373343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373331 5978 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:19:42.373374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373362 5978 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:19:42.373497 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373487 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:19:42.373535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373503 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:19:42.373535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373509 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:19:42.373535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373515 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:19:42.373535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373520 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:19:42.373535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373527 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:19:42.373535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373532 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:19:42.373805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373539 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:19:42.373805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373546 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:19:42.373805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373553 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:19:42.373805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373567 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:19:42.373805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373576 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:19:42.373805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373605 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:19:42.373805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.373610 5978 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:19:42.377151 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.377137 5978 server.go:1263] "Started kubelet" Oct 31 00:19:42.377237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.377209 5978 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:19:42.377314 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.377235 5978 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:19:42.377366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.377358 5978 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:19:42.384027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.377699 5978 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:19:42.384027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.378598 5978 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:19:42.377746 ip-10-0-129-199 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:19:42.384265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.384217 5978 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:19:42.384395 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.384352 5978 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:19:42.385178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.384981 5978 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:13:33 +0000 UTC" deadline="2028-06-16 04:23:36.602920674 +0000 UTC" Oct 31 00:19:42.385178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.385021 5978 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14260h3m54.217902902s" Oct 31 00:19:42.385178 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:42.385054 5978 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-199.ec2.internal\" not found" Oct 31 00:19:42.385178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.385083 5978 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:19:42.385178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.385089 5978 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:19:42.385178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.385136 5978 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:19:42.387401 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.387380 5978 factory.go:55] Registering systemd factory Oct 31 00:19:42.387401 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.387401 5978 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388739 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e75b3b-ffb1-4cda-9203-f4882bd385e7" volumeName="kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-tuned" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388780 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fa8cd46-8095-4e93-b23a-0a422a47eab3" volumeName="kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388794 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a1f30-b435-43fd-9f63-a1123bd6e661" volumeName="kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388808 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dae54b3-dceb-4795-85ec-42f9fb9ff256" volumeName="kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388820 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0bd8aa-5627-4ece-8ef4-97a529ab89ae" volumeName="kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388830 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="364a8716-433d-4181-94ed-f6ccd33a0828" volumeName="kubernetes.io/projected/364a8716-433d-4181-94ed-f6ccd33a0828-kube-api-access-6m2pd" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388844 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6249a4c3-cd70-4c06-aa16-9864e6493086" volumeName="kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388855 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388868 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388880 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dae54b3-dceb-4795-85ec-42f9fb9ff256" volumeName="kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388892 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0bd8aa-5627-4ece-8ef4-97a529ab89ae" volumeName="kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-client-ca-bundle" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388904 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388915 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6f8e29e-fb86-4153-b168-0bb1743e44cd" volumeName="kubernetes.io/configmap/d6f8e29e-fb86-4153-b168-0bb1743e44cd-config-volume" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388926 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="364a8716-433d-4181-94ed-f6ccd33a0828" volumeName="kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388942 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465a3984-adbc-4677-b22f-9e9240a770f8" volumeName="kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-metrics-client-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388953 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465a3984-adbc-4677-b22f-9e9240a770f8" volumeName="kubernetes.io/empty-dir/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-textfile" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388965 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a1f30-b435-43fd-9f63-a1123bd6e661" volumeName="kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388977 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b1af480-ae8f-483e-bd52-947094fcc3ff" volumeName="kubernetes.io/projected/2b1af480-ae8f-483e-bd52-947094fcc3ff-kube-api-access-jfq6z" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.388988 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389000 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb59038-b544-4cb1-a2af-ded4bb5d6431" volumeName="kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389029 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dae54b3-dceb-4795-85ec-42f9fb9ff256" volumeName="kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389042 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="064705a9-54fd-4528-ae88-1cb4596c5b6e" volumeName="kubernetes.io/secret/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389054 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb59038-b544-4cb1-a2af-ded4bb5d6431" volumeName="kubernetes.io/empty-dir/2cb59038-b544-4cb1-a2af-ded4bb5d6431-tmp" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389070 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb59038-b544-4cb1-a2af-ded4bb5d6431" volumeName="kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389081 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="364a8716-433d-4181-94ed-f6ccd33a0828" volumeName="kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389093 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a1f30-b435-43fd-9f63-a1123bd6e661" volumeName="kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389104 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abe218cc-f275-4474-a54e-3ba513e969c2" volumeName="kubernetes.io/projected/abe218cc-f275-4474-a54e-3ba513e969c2-kube-api-access-khv2q" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389122 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="064705a9-54fd-4528-ae88-1cb4596c5b6e" volumeName="kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389139 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="364a8716-433d-4181-94ed-f6ccd33a0828" volumeName="kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389150 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465a3984-adbc-4677-b22f-9e9240a770f8" volumeName="kubernetes.io/projected/465a3984-adbc-4677-b22f-9e9240a770f8-kube-api-access-rwv2b" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389161 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389172 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389183 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a719455f-b7a1-4394-8258-6d84f4c0ccd3" volumeName="kubernetes.io/secret/a719455f-b7a1-4394-8258-6d84f4c0ccd3-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389194 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389205 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb59038-b544-4cb1-a2af-ded4bb5d6431" volumeName="kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389218 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fa8cd46-8095-4e93-b23a-0a422a47eab3" volumeName="kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389229 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d60169a-68c9-4cfd-bbd7-73ecac659695" volumeName="kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389241 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abe218cc-f275-4474-a54e-3ba513e969c2" volumeName="kubernetes.io/empty-dir/abe218cc-f275-4474-a54e-3ba513e969c2-data-volume" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389253 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6f8e29e-fb86-4153-b168-0bb1743e44cd" volumeName="kubernetes.io/secret/d6f8e29e-fb86-4153-b168-0bb1743e44cd-metrics-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389266 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0bd8aa-5627-4ece-8ef4-97a529ab89ae" volumeName="kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389276 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389288 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb59038-b544-4cb1-a2af-ded4bb5d6431" volumeName="kubernetes.io/projected/2cb59038-b544-4cb1-a2af-ded4bb5d6431-kube-api-access-2hphs" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389300 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="364a8716-433d-4181-94ed-f6ccd33a0828" volumeName="kubernetes.io/empty-dir/364a8716-433d-4181-94ed-f6ccd33a0828-volume-directive-shadow" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389311 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a38f09d-12b6-481c-bb13-c686226854a6" volumeName="kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389323 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389337 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abe218cc-f275-4474-a54e-3ba513e969c2" volumeName="kubernetes.io/secret/abe218cc-f275-4474-a54e-3ba513e969c2-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389363 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389375 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56c984b-c9d9-4d3f-86fd-2314f01a6673" volumeName="kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389387 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b70404c4-e809-42c9-9efc-d6262e52815c" volumeName="kubernetes.io/projected/b70404c4-e809-42c9-9efc-d6262e52815c-kube-api-access-2sk42" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389398 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f99f8918-3e3d-456b-be91-db4120b67424" volumeName="kubernetes.io/projected/f99f8918-3e3d-456b-be91-db4120b67424-kube-api-access-76qh9" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389410 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fa8cd46-8095-4e93-b23a-0a422a47eab3" volumeName="kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389422 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389433 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389493 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389511 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a38f09d-12b6-481c-bb13-c686226854a6" volumeName="kubernetes.io/projected/8a38f09d-12b6-481c-bb13-c686226854a6-kube-api-access-drh6b" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389522 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389545 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dae54b3-dceb-4795-85ec-42f9fb9ff256" volumeName="kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389557 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389568 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465a3984-adbc-4677-b22f-9e9240a770f8" volumeName="kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389579 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a1f30-b435-43fd-9f63-a1123bd6e661" volumeName="kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389590 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389601 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d60169a-68c9-4cfd-bbd7-73ecac659695" volumeName="kubernetes.io/projected/2d60169a-68c9-4cfd-bbd7-73ecac659695-kube-api-access-2v5nl" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389612 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af6b8f1-b029-4ae2-99cb-df0315daa268" volumeName="kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-cni-binary-copy" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389624 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dd2bec9-6796-4e26-ad19-bfd15842852b" volumeName="kubernetes.io/projected/7dd2bec9-6796-4e26-ad19-bfd15842852b-kube-api-access-nkrbb" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389635 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abe218cc-f275-4474-a54e-3ba513e969c2" volumeName="kubernetes.io/configmap/abe218cc-f275-4474-a54e-3ba513e969c2-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389646 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389657 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f99f8918-3e3d-456b-be91-db4120b67424" volumeName="kubernetes.io/configmap/f99f8918-3e3d-456b-be91-db4120b67424-serviceca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389669 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389684 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389697 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a1f30-b435-43fd-9f63-a1123bd6e661" volumeName="kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389710 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b70404c4-e809-42c9-9efc-d6262e52815c" volumeName="kubernetes.io/configmap/b70404c4-e809-42c9-9efc-d6262e52815c-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389721 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="064705a9-54fd-4528-ae88-1cb4596c5b6e" volumeName="kubernetes.io/projected/064705a9-54fd-4528-ae88-1cb4596c5b6e-kube-api-access-j8rbb" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389732 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb59038-b544-4cb1-a2af-ded4bb5d6431" volumeName="kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389742 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af6b8f1-b029-4ae2-99cb-df0315daa268" volumeName="kubernetes.io/projected/3af6b8f1-b029-4ae2-99cb-df0315daa268-kube-api-access-5dlwf" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389754 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57df0df3-a039-45eb-8ae2-02433f6b0b2e" volumeName="kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389766 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a1f30-b435-43fd-9f63-a1123bd6e661" volumeName="kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389776 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af6b8f1-b029-4ae2-99cb-df0315daa268" volumeName="kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-daemon-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389790 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389801 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dae54b3-dceb-4795-85ec-42f9fb9ff256" volumeName="kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389812 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0bd8aa-5627-4ece-8ef4-97a529ab89ae" volumeName="kubernetes.io/empty-dir/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-audit-log" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389823 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fa8cd46-8095-4e93-b23a-0a422a47eab3" volumeName="kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389835 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fa8cd46-8095-4e93-b23a-0a422a47eab3" volumeName="kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389847 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb59038-b544-4cb1-a2af-ded4bb5d6431" volumeName="kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389858 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fa8cd46-8095-4e93-b23a-0a422a47eab3" volumeName="kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389870 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="364a8716-433d-4181-94ed-f6ccd33a0828" volumeName="kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-metrics-client-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389883 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a1f30-b435-43fd-9f63-a1123bd6e661" volumeName="kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389895 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6f8e29e-fb86-4153-b168-0bb1743e44cd" volumeName="kubernetes.io/empty-dir/d6f8e29e-fb86-4153-b168-0bb1743e44cd-tmp-dir" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389906 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e75b3b-ffb1-4cda-9203-f4882bd385e7" volumeName="kubernetes.io/projected/05e75b3b-ffb1-4cda-9203-f4882bd385e7-kube-api-access-tfzq2" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389918 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389932 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389943 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47350ce3-ba5e-4393-ac0f-653b41bc28fd" volumeName="kubernetes.io/empty-dir/47350ce3-ba5e-4393-ac0f-653b41bc28fd-tmp-dir" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389954 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389966 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dae54b3-dceb-4795-85ec-42f9fb9ff256" volumeName="kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389979 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb59038-b544-4cb1-a2af-ded4bb5d6431" volumeName="kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.389991 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a38f09d-12b6-481c-bb13-c686226854a6" volumeName="kubernetes.io/configmap/8a38f09d-12b6-481c-bb13-c686226854a6-metrics-client-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390002 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6f8e29e-fb86-4153-b168-0bb1743e44cd" volumeName="kubernetes.io/projected/d6f8e29e-fb86-4153-b168-0bb1743e44cd-kube-api-access-7kl9k" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390013 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390024 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390035 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="064705a9-54fd-4528-ae88-1cb4596c5b6e" volumeName="kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-env-overrides" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390047 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="064705a9-54fd-4528-ae88-1cb4596c5b6e" volumeName="kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390059 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390084 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dd2bec9-6796-4e26-ad19-bfd15842852b" volumeName="kubernetes.io/secret/7dd2bec9-6796-4e26-ad19-bfd15842852b-cert" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390095 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fa8cd46-8095-4e93-b23a-0a422a47eab3" volumeName="kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390106 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390117 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d60169a-68c9-4cfd-bbd7-73ecac659695" volumeName="kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390130 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0bd8aa-5627-4ece-8ef4-97a529ab89ae" volumeName="kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390141 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0bd8aa-5627-4ece-8ef4-97a529ab89ae" volumeName="kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390152 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390163 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e75b3b-ffb1-4cda-9203-f4882bd385e7" volumeName="kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-tmp" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390176 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dae54b3-dceb-4795-85ec-42f9fb9ff256" volumeName="kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390188 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dae54b3-dceb-4795-85ec-42f9fb9ff256" volumeName="kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390200 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465a3984-adbc-4677-b22f-9e9240a770f8" volumeName="kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390228 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390239 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a719455f-b7a1-4394-8258-6d84f4c0ccd3" volumeName="kubernetes.io/projected/a719455f-b7a1-4394-8258-6d84f4c0ccd3-kube-api-access-mr978" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390250 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0bd8aa-5627-4ece-8ef4-97a529ab89ae" volumeName="kubernetes.io/projected/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-kube-api-access-s4c5s" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390261 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb59038-b544-4cb1-a2af-ded4bb5d6431" volumeName="kubernetes.io/configmap/2cb59038-b544-4cb1-a2af-ded4bb5d6431-metrics-client-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390272 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57df0df3-a039-45eb-8ae2-02433f6b0b2e" volumeName="kubernetes.io/projected/57df0df3-a039-45eb-8ae2-02433f6b0b2e-kube-api-access-8x95q" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390284 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390295 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390307 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13ec7561-c654-4156-9ec3-5cb5893881b8" volumeName="kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390318 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d60169a-68c9-4cfd-bbd7-73ecac659695" volumeName="kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-binary-copy" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390329 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465a3984-adbc-4677-b22f-9e9240a770f8" volumeName="kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390341 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a1f30-b435-43fd-9f63-a1123bd6e661" volumeName="kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390353 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a38f09d-12b6-481c-bb13-c686226854a6" volumeName="kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390365 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90ede307-5582-4953-abaa-25d4393dc55c" volumeName="kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390376 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f9f082d-9176-491a-8d5e-281ee6665dae" volumeName="kubernetes.io/configmap/1f9f082d-9176-491a-8d5e-281ee6665dae-konnectivity-ca" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390388 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f9f082d-9176-491a-8d5e-281ee6665dae" volumeName="kubernetes.io/secret/1f9f082d-9176-491a-8d5e-281ee6665dae-agent-certs" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390399 5978 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47350ce3-ba5e-4393-ac0f-653b41bc28fd" volumeName="kubernetes.io/projected/47350ce3-ba5e-4393-ac0f-653b41bc28fd-kube-api-access-flpkx" seLinuxMountContext="" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390409 5978 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:19:42.391266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.390416 5978 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:19:42.397215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.391433 5978 factory.go:153] Registering CRI-O factory Oct 31 00:19:42.397215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.391470 5978 factory.go:223] Registration of the crio container factory successfully Oct 31 00:19:42.397215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.391879 5978 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:19:42.397215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.391908 5978 factory.go:103] Registering Raw factory Oct 31 00:19:42.397215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.391920 5978 manager.go:1199] Started watching for new ooms in manager Oct 31 00:19:42.397215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.392420 5978 manager.go:319] Starting recovery of all containers Oct 31 00:19:42.408344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.408296 5978 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:19:42.416875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.416844 5978 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:19:42.416875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.416872 5978 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:19:42.417107 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.416928 5978 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:19:42.417107 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:42.416976 5978 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:19:42.430376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.430344 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:19:42.432518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.432438 5978 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:19:42.434332 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.434301 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:19:42.434750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.434725 5978 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:19:42.436219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.436187 5978 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:19:42.442699 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.442670 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:19:42.443054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.443027 5978 generic.go:356] "Generic (PLEG): container finished" podID="064705a9-54fd-4528-ae88-1cb4596c5b6e" containerID="eedb5073e0ea0c851e399952ddb84421cd92c5ee6281b0a40d3bad25951455ba" exitCode=1 Oct 31 00:19:42.449272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.449239 5978 generic.go:356] "Generic (PLEG): container finished" podID="465a3984-adbc-4677-b22f-9e9240a770f8" containerID="8dca9a5de6099dcdfb08a7b240b416b5ac61810caedc53dd45eae1a140871bbb" exitCode=0 Oct 31 00:19:42.456792 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.456759 5978 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:19:42.456994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.456973 5978 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:19:42.457066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.457035 5978 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="c1c6d4c2f0073098bfee4185dc530e40c35e8378c08223303414dd8486ca4f6e" exitCode=0 Oct 31 00:19:42.457066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.457053 5978 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="becf88363f160ddc4d838ca7a88728489a7638a8c8485bc416e3fdefb90ff936" exitCode=0 Oct 31 00:19:42.457066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.457059 5978 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="619c0b2273b3273804962b179ab5ba39603c66368fb711a5855e4f46632e38e6" exitCode=0 Oct 31 00:19:42.457066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.457065 5978 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="75204a122fd669d223dc818da5588c4c345e1b1d953f79be19a38adb3eb5ee42" exitCode=0 Oct 31 00:19:42.457231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.457071 5978 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="85eb71e41ab9d186337a379ac56c4ea08c2969d56d8bf1fcf2acacf634b0637f" exitCode=0 Oct 31 00:19:42.457231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.457076 5978 generic.go:356] "Generic (PLEG): container finished" podID="2d60169a-68c9-4cfd-bbd7-73ecac659695" containerID="2080f43e4022d6e03623d14b050bd3e71ef95da20a1de21e8b1e2f7f52cc937a" exitCode=0 Oct 31 00:19:42.471799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.471769 5978 generic.go:356] "Generic (PLEG): container finished" podID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerID="f94e4d0b68df9df554ee8f6a4eec3a65d85d4d43ac8df05dd72a3b0c381191cc" exitCode=0 Oct 31 00:19:42.476960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.476885 5978 generic.go:356] "Generic (PLEG): container finished" podID="d3b869eb4ef8b2ca62aee88ee86a131f" containerID="0ef232731fce729f61bc4a971a4c314d979c8a05da8d54d6fc7b34cc376a2ff1" exitCode=0 Oct 31 00:19:42.482249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.482214 5978 generic.go:356] "Generic (PLEG): container finished" podID="90ede307-5582-4953-abaa-25d4393dc55c" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" exitCode=0 Oct 31 00:19:42.517183 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:42.517149 5978 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 00:19:42.528939 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.528909 5978 manager.go:324] Recovery completed Oct 31 00:19:42.556601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556574 5978 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:19:42.556601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556593 5978 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:19:42.556799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556618 5978 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:19:42.556799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556755 5978 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:19:42.556799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556766 5978 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:19:42.556799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556784 5978 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:19:42.556799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556789 5978 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:19:42.556799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556795 5978 policy_none.go:50] "Start" Oct 31 00:19:42.556954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556804 5978 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:19:42.556954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.556813 5978 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:19:42.558383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.558366 5978 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:19:42.558422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.558383 5978 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:19:42.558422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.558391 5978 policy_none.go:44] "Start" Oct 31 00:19:42.565259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.565221 5978 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:19:42.568042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.568026 5978 manager.go:341] "Starting Device Plugin manager" Oct 31 00:19:42.568685 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:42.568057 5978 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:19:42.568747 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.568689 5978 server.go:87] "Starting device plugin registration server" Oct 31 00:19:42.568974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.568963 5978 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:19:42.569005 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.568978 5978 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:19:42.569101 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.569074 5978 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:19:42.569234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.569197 5978 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:19:42.569284 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.569250 5978 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:19:42.569330 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.569311 5978 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:19:42.569464 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.569376 5978 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:19:42.569233015Z","UUID":"98bc9e48-6983-4426-98e2-6cb639e4d7ed","Handler":null,"Name":"","Endpoint":""} Oct 31 00:19:42.570603 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.570585 5978 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:19:42.570705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.570614 5978 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:19:42.571333 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:42.571311 5978 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:19:42.670276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.670244 5978 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:19:42.672163 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.672131 5978 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientMemory" Oct 31 00:19:42.672298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.672172 5978 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 00:19:42.672298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.672187 5978 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-199.ec2.internal" event="NodeHasSufficientPID" Oct 31 00:19:42.672298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.672239 5978 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.717883 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.717808 5978 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal"] Oct 31 00:19:42.718620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.718574 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" event={"ID":"ad1af2674e0c55384c949adc8f5b6f9c","Type":"ContainerStarted","Data":"9cdeeb37adfa2716df0a9041df3a10c4048a6436988ff3b5626b72d131d542c4"} Oct 31 00:19:42.718620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.718606 5978 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.718620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.718620 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" event={"ID":"ad1af2674e0c55384c949adc8f5b6f9c","Type":"ContainerStarted","Data":"f95f9e90756a204a140abb4e56732e931cda0f25c18a8d2ba3d504610a3c90df"} Oct 31 00:19:42.719009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.718693 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"d3b869eb4ef8b2ca62aee88ee86a131f","Type":"ContainerStarted","Data":"28345d3584d1284e590625a134a2e6c1da184bbcc8e5f957f0ffef51a02aac34"} Oct 31 00:19:42.719009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.718708 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"d3b869eb4ef8b2ca62aee88ee86a131f","Type":"ContainerDied","Data":"0ef232731fce729f61bc4a971a4c314d979c8a05da8d54d6fc7b34cc376a2ff1"} Oct 31 00:19:42.719009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.718720 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" event={"ID":"d3b869eb4ef8b2ca62aee88ee86a131f","Type":"ContainerStarted","Data":"feae4182e7d2d731c263c8b3ddfa1e81e707fef6583ba1cb18aa78ed7682c00f"} Oct 31 00:19:42.719009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.718786 5978 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.763859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.763785 5978 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.763859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.763859 5978 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.771663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.771636 5978 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:19:42.771799 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:42.771721 5978 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.834056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.834012 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.834056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.834056 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.834261 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.834074 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad1af2674e0c55384c949adc8f5b6f9c-config\") pod \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" (UID: \"ad1af2674e0c55384c949adc8f5b6f9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.871985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.871949 5978 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:19:42.872125 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:42.872006 5978 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.934382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.934345 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.934542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.934387 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad1af2674e0c55384c949adc8f5b6f9c-config\") pod \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" (UID: \"ad1af2674e0c55384c949adc8f5b6f9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.934542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.934415 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.934542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.934475 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.934542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.934502 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad1af2674e0c55384c949adc8f5b6f9c-config\") pod \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" (UID: \"ad1af2674e0c55384c949adc8f5b6f9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:19:42.934542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:42.934490 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3b869eb4ef8b2ca62aee88ee86a131f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" (UID: \"d3b869eb4ef8b2ca62aee88ee86a131f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:43.368841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.368803 5978 apiserver.go:52] "Watching apiserver" Oct 31 00:19:43.410882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.410846 5978 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:19:43.411649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.411621 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-gffpv","openshift-monitoring/prometheus-k8s-0","openshift-insights/insights-runtime-extractor-qg2xf","openshift-dns/node-resolver-npncc","openshift-multus/multus-bfsdv","openshift-network-diagnostics/network-check-target-mswcp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal","openshift-network-operator/iptables-alerter-b2slc","openshift-console/console-69b4c98b6b-w7lch","openshift-dns/dns-default-nqpmv","kube-system/konnectivity-agent-bshvl","openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr","openshift-multus/multus-additional-cni-plugins-mhppg","openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf","kube-system/global-pull-secret-syncer-z4hfm","openshift-console/console-5dcdc4b64f-p6xjm","openshift-ingress-canary/ingress-canary-kqx7w","openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf","openshift-ovn-kubernetes/ovnkube-node-sgkkv","openshift-multus/network-metrics-daemon-xzjcg","kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal","openshift-monitoring/monitoring-plugin-77f4f76594-br9bn","openshift-monitoring/node-exporter-lhrff","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f","openshift-console/console-fb49695f-qnzhd","openshift-image-registry/node-ca-jpxt2","openshift-monitoring/alertmanager-main-0","openshift-monitoring/metrics-server-6cbf58948-7xm9f"] Oct 31 00:19:43.446171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.446134 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:19:43.446396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.446383 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6b92m\"" Oct 31 00:19:43.446528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.446388 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dsrzd\"" Oct 31 00:19:43.446528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.446385 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jw4zk\"" Oct 31 00:19:43.446528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.446427 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgmpl\"" Oct 31 00:19:43.446528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.446476 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:19:43.456431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.456410 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.456515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.456486 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.458203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458178 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.458349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458266 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.458349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458270 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:19:43.458349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458291 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.458525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458394 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.458525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458432 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.458525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458434 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.458525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458438 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.458708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458568 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.458708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458581 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:19:43.458708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458589 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.458708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458596 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.458708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458626 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.458708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458626 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.458708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458659 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.458708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458678 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:19:43.458919 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.458737 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:19:43.467204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467168 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:19:43.467413 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467374 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wlc7\"" Oct 31 00:19:43.467515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467401 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rqjpt\"" Oct 31 00:19:43.467515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467423 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:19:43.467515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467441 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-grldd\"" Oct 31 00:19:43.467515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467436 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dqh4t\"" Oct 31 00:19:43.467515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467471 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:19:43.467515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467514 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:19:43.467720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467541 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:19:43.467720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467567 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4z9hp\"" Oct 31 00:19:43.467870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.467855 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.468865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.468851 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:19:43.468956 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.468948 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:19:43.469055 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469043 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.469224 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469210 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.469330 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469316 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:19:43.469418 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469405 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:19:43.469529 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469514 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:19:43.469576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469545 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.469717 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469705 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:19:43.469830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469820 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f62jd\"" Oct 31 00:19:43.469909 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469895 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:19:43.469965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469897 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:19:43.469965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469916 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:19:43.469965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469940 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:19:43.470087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469981 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rltkk\"" Oct 31 00:19:43.470087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.469998 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:19:43.470087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.470025 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l44c8\"" Oct 31 00:19:43.488306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.488267 5978 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:19:43.499289 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.499263 5978 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:43.499425 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.499351 5978 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:19:43.526188 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.526162 5978 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:19:43.526282 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:43.526217 5978 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-199.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-199.ec2.internal" Oct 31 00:19:43.532406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.532383 5978 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:19:43.532489 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:43.532428 5978 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-199.ec2.internal" Oct 31 00:19:43.538757 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.538733 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrbb\" (UniqueName: \"kubernetes.io/projected/7dd2bec9-6796-4e26-ad19-bfd15842852b-kube-api-access-nkrbb\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:43.538807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.538768 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.538841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.538806 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x95q\" (UniqueName: \"kubernetes.io/projected/57df0df3-a039-45eb-8ae2-02433f6b0b2e-kube-api-access-8x95q\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:43.538841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.538837 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.538917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.538878 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.538976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.538963 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-kubelet-config\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:43.539023 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.538985 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-dbus\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:43.539023 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539002 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.539115 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539059 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:43.539160 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539113 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.539192 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539152 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-tuned\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.539234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539190 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-grpc-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.539234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539209 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abe218cc-f275-4474-a54e-3ba513e969c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.539234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539219 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-tuned\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.539333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539266 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.539333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539296 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-tls\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.539333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539323 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-tls\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.539469 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539346 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.539469 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539380 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a56c984b-c9d9-4d3f-86fd-2314f01a6673-original-pull-secret\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:43.539469 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539410 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47350ce3-ba5e-4393-ac0f-653b41bc28fd-tmp-dir\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:19:43.539595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539473 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.539595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539473 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.539595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539513 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47350ce3-ba5e-4393-ac0f-653b41bc28fd-tmp-dir\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:19:43.539595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539524 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.539595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539545 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-etc-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.539595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539585 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb59038-b544-4cb1-a2af-ded4bb5d6431-metrics-client-ca\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.539819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539604 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/3af6b8f1-b029-4ae2-99cb-df0315daa268-kube-api-access-5dlwf\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.539819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539624 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719455f-b7a1-4394-8258-6d84f4c0ccd3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:43.539819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539639 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.539819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539659 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.539819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539692 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-sys-fs\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.539819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539735 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99f8918-3e3d-456b-be91-db4120b67424-serviceca\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:19:43.539819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539769 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-node-log\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.539819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539792 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cb59038-b544-4cb1-a2af-ded4bb5d6431-tmp\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.539819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539808 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abe218cc-f275-4474-a54e-3ba513e969c2-crio-socket\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539829 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abe218cc-f275-4474-a54e-3ba513e969c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539859 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a719455f-b7a1-4394-8258-6d84f4c0ccd3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539873 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539898 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539907 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cb59038-b544-4cb1-a2af-ded4bb5d6431-tmp\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539921 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99f8918-3e3d-456b-be91-db4120b67424-host\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539947 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-systemd\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.539973 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dd2bec9-6796-4e26-ad19-bfd15842852b-cert\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540016 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drh6b\" (UniqueName: \"kubernetes.io/projected/8a38f09d-12b6-481c-bb13-c686226854a6-kube-api-access-drh6b\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540045 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540072 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540091 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abe218cc-f275-4474-a54e-3ba513e969c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540097 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540127 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540143 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-socket-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540170 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-conf-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540196 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr978\" (UniqueName: \"kubernetes.io/projected/a719455f-b7a1-4394-8258-6d84f4c0ccd3-kube-api-access-mr978\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:43.540200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540200 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540223 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540259 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540291 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-sys\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540298 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540319 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hphs\" (UniqueName: \"kubernetes.io/projected/2cb59038-b544-4cb1-a2af-ded4bb5d6431-kube-api-access-2hphs\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540345 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-bin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540371 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540397 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540422 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540448 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-tmp\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540500 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-netns\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540511 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99f8918-3e3d-456b-be91-db4120b67424-serviceca\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540515 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-ovn\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540544 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-netd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540582 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05e75b3b-ffb1-4cda-9203-f4882bd385e7-tmp\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540586 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-cnibin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540598 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540625 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540662 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540721 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540740 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flpkx\" (UniqueName: \"kubernetes.io/projected/47350ce3-ba5e-4393-ac0f-653b41bc28fd-kube-api-access-flpkx\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540759 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f8e29e-fb86-4153-b168-0bb1743e44cd-tmp-dir\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540779 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb89\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540823 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-kubernetes\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.540837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540839 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-system-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540856 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a38f09d-12b6-481c-bb13-c686226854a6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540874 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540884 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f8e29e-fb86-4153-b168-0bb1743e44cd-tmp-dir\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540882 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540924 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540942 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.540977 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rbb\" (UniqueName: \"kubernetes.io/projected/064705a9-54fd-4528-ae88-1cb4596c5b6e-kube-api-access-j8rbb\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541073 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541119 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541169 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-wtmp\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541196 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-metrics-client-ca\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541223 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541276 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541348 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541381 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5nl\" (UniqueName: \"kubernetes.io/projected/2d60169a-68c9-4cfd-bbd7-73ecac659695-kube-api-access-2v5nl\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541431 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541466 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-netns\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541486 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-multus\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541504 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-textfile\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541519 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541542 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-log-socket\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541564 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541581 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnvmn\" (UniqueName: \"kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541581 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-textfile\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541597 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541613 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541644 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541667 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq6z\" (UniqueName: \"kubernetes.io/projected/2b1af480-ae8f-483e-bd52-947094fcc3ff-kube-api-access-jfq6z\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541684 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-systemd-units\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541730 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-systemd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541763 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541793 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541837 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.541887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541872 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541925 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-kubelet\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541954 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-root\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.541983 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542012 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f8e29e-fb86-4153-b168-0bb1743e44cd-config-volume\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542032 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542049 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-config\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542065 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47350ce3-ba5e-4393-ac0f-653b41bc28fd-hosts-file\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542088 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-cnibin\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542106 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl9k\" (UniqueName: \"kubernetes.io/projected/d6f8e29e-fb86-4153-b168-0bb1743e44cd-kube-api-access-7kl9k\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542115 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542121 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-device-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542166 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/364a8716-433d-4181-94ed-f6ccd33a0828-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542190 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f8e29e-fb86-4153-b168-0bb1743e44cd-config-volume\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542221 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542240 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542257 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542260 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/364a8716-433d-4181-94ed-f6ccd33a0828-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542271 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f8e29e-fb86-4153-b168-0bb1743e44cd-metrics-tls\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542277 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-config\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542295 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-kubelet\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542333 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542367 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-hostroot\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542396 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzj5s\" (UniqueName: \"kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542420 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542428 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542478 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542514 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542541 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-lib-modules\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542567 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542568 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6f8e29e-fb86-4153-b168-0bb1743e44cd-metrics-tls\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542541 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542672 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542708 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542736 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abe218cc-f275-4474-a54e-3ba513e969c2-data-volume\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542818 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abe218cc-f275-4474-a54e-3ba513e969c2-data-volume\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542940 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542941 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-sys\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.542991 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwv2b\" (UniqueName: \"kubernetes.io/projected/465a3984-adbc-4677-b22f-9e9240a770f8-kube-api-access-rwv2b\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543020 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-run\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543050 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543089 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543091 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543111 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543119 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543124 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543108 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovn-node-metrics-cert\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543159 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543172 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543175 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543217 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543238 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543246 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-client-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543242 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543279 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543293 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543318 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543327 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543357 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qh9\" (UniqueName: \"kubernetes.io/projected/f99f8918-3e3d-456b-be91-db4120b67424-kube-api-access-76qh9\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543374 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543383 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzq2\" (UniqueName: \"kubernetes.io/projected/05e75b3b-ffb1-4cda-9203-f4882bd385e7-kube-api-access-tfzq2\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543395 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543403 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543408 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovn-node-metrics-cert\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.543381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543447 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-client-certs\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543500 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-conf\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543543 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-host\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543570 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543580 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543689 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-daemon-config\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543707 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-client-certs\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543746 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543760 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543810 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-metrics-server-audit-profiles\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543823 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543827 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543853 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543873 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-modprobe-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543818 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543890 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-script-lib\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543901 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543935 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-multus-certs\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.543982 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544061 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544070 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysconfig\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544195 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f9f082d-9176-491a-8d5e-281ee6665dae-agent-certs\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544223 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544249 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-k8s-cni-cncf-io\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544303 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544303 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwthq\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544407 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f9f082d-9176-491a-8d5e-281ee6665dae-agent-certs\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544437 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sk42\" (UniqueName: \"kubernetes.io/projected/b70404c4-e809-42c9-9efc-d6262e52815c-kube-api-access-2sk42\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544493 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f082d-9176-491a-8d5e-281ee6665dae-konnectivity-ca\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544528 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544497 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/364a8716-433d-4181-94ed-f6ccd33a0828-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544557 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-audit-log\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544591 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544621 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544694 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544727 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-audit-log\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544766 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544700 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f082d-9176-491a-8d5e-281ee6665dae-konnectivity-ca\") pod \"konnectivity-agent-bshvl\" (UID: \"1f9f082d-9176-491a-8d5e-281ee6665dae\") " pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544792 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544735 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-socket-dir-parent\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544823 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5c7xv\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544830 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544852 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c5s\" (UniqueName: \"kubernetes.io/projected/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-kube-api-access-s4c5s\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544943 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544965 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b70404c4-e809-42c9-9efc-d6262e52815c-host-slash\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544875 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9vtvc\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544880 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544991 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545009 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545057 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.544870 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545132 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-registration-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545137 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545149 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545148 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mg4tg\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545181 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v4wl8\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545185 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545238 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-var-lib-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545256 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pk56l\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545271 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cno706g3ng8gr\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545288 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vhm6s\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545296 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545317 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-system-cni-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545340 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545338 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khv2q\" (UniqueName: \"kubernetes.io/projected/abe218cc-f275-4474-a54e-3ba513e969c2-kube-api-access-khv2q\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545378 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b70404c4-e809-42c9-9efc-d6262e52815c-iptables-alerter-script\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545397 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545496 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545530 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-env-overrides\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545561 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545576 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b70404c4-e809-42c9-9efc-d6262e52815c-iptables-alerter-script\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545586 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-slash\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545614 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545651 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545678 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-var-lib-kubelet\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545680 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-env-overrides\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545733 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-bin\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545764 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0df3-a039-45eb-8ae2-02433f6b0b2e-metrics-certs\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545776 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2pd\" (UniqueName: \"kubernetes.io/projected/364a8716-433d-4181-94ed-f6ccd33a0828-kube-api-access-6m2pd\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545810 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-cni-binary-copy\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545830 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-etc-kubernetes\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545848 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.545865 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhtf\" (UniqueName: \"kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546000 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-cni-binary-copy\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.545967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546021 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546096 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546647 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546651 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e3j1gt3835o86\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546722 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546761 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546761 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9wq7f\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546763 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4w2kf\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546799 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pp79q\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546879 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9h6ltk5ck35h9\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546891 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.546985 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.547056 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.547251 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.547257 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.547276 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.547363 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.547387 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.547401 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:19:43.548404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.547478 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.549596 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.549574 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abe218cc-f275-4474-a54e-3ba513e969c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.549596 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.549589 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.549742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.549601 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.549892 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.549608 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-tls\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.549962 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.549923 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.550026 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.549992 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-secret-metrics-server-tls\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.550076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.550053 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.550076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.550067 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-grpc-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.550150 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.550057 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.550150 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.550086 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.550236 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dd2bec9-6796-4e26-ad19-bfd15842852b-cert\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.550242 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.550430 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.550537 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.550805 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.553403 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.553636 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.553803 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a38f09d-12b6-481c-bb13-c686226854a6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.553974 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.554194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554147 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d60169a-68c9-4cfd-bbd7-73ecac659695-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.554733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554325 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.554733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554398 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.554733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554375 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.554733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554419 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-client-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.554733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554629 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.554733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554658 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-metrics-server-audit-profiles\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.554733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554696 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.554733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554697 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.554733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554716 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.555105 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554755 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-tls\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.555105 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.554856 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.555273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.555253 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.555619 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.555600 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.555721 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.555707 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.555872 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.555856 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.556206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.556191 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:19:43.556597 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.556578 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:19:43.557930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.557908 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:19:43.558019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.557935 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:19:43.558019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.557946 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9prwf\"" Oct 31 00:19:43.558686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.558673 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:19:43.559780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.559762 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.559927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.559895 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb59038-b544-4cb1-a2af-ded4bb5d6431-metrics-client-ca\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.561105 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561088 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:19:43.561105 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561099 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:19:43.561205 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561091 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:19:43.561205 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561121 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:19:43.561205 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561122 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a38f09d-12b6-481c-bb13-c686226854a6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:43.561404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561392 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:19:43.561525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561508 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 00:19:43.561572 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561547 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.561666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561624 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/465a3984-adbc-4677-b22f-9e9240a770f8-metrics-client-ca\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.561731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.561686 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.562048 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562021 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.562138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562067 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.562682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562442 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:19:43.562682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562468 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:19:43.562682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562477 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jhm6q\"" Oct 31 00:19:43.562682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562502 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:19:43.562682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562512 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p7sw8\"" Oct 31 00:19:43.562682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562541 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.562682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562643 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364a8716-433d-4181-94ed-f6ccd33a0828-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.562682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562659 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:19:43.562955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562739 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.562955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562796 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.563015 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.562953 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.563091 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.563076 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2cb59038-b544-4cb1-a2af-ded4bb5d6431-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.563734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.563720 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.563766 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.563726 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.564897 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.564880 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.564936 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.564886 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-daemon-config\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.564965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.564939 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/064705a9-54fd-4528-ae88-1cb4596c5b6e-ovnkube-script-lib\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.566176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.566153 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.566239 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.566183 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.570695 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.570674 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rbb\" (UniqueName: \"kubernetes.io/projected/064705a9-54fd-4528-ae88-1cb4596c5b6e-kube-api-access-j8rbb\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.570841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.570823 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.572488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.572471 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.574164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.574148 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flpkx\" (UniqueName: \"kubernetes.io/projected/47350ce3-ba5e-4393-ac0f-653b41bc28fd-kube-api-access-flpkx\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:19:43.574829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.574809 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrbb\" (UniqueName: \"kubernetes.io/projected/7dd2bec9-6796-4e26-ad19-bfd15842852b-kube-api-access-nkrbb\") pod \"ingress-canary-kqx7w\" (UID: \"7dd2bec9-6796-4e26-ad19-bfd15842852b\") " pod="openshift-ingress-canary/ingress-canary-kqx7w" Oct 31 00:19:43.574906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.574811 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/3af6b8f1-b029-4ae2-99cb-df0315daa268-kube-api-access-5dlwf\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.574971 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.574953 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr978\" (UniqueName: \"kubernetes.io/projected/a719455f-b7a1-4394-8258-6d84f4c0ccd3-kube-api-access-mr978\") pod \"monitoring-plugin-77f4f76594-br9bn\" (UID: \"a719455f-b7a1-4394-8258-6d84f4c0ccd3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:43.575996 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.575979 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgxt\" (UniqueName: \"kubernetes.io/projected/6249a4c3-cd70-4c06-aa16-9864e6493086-kube-api-access-chgxt\") pod \"network-check-target-mswcp\" (UID: \"6249a4c3-cd70-4c06-aa16-9864e6493086\") " pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:43.577109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.577092 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hphs\" (UniqueName: \"kubernetes.io/projected/2cb59038-b544-4cb1-a2af-ded4bb5d6431-kube-api-access-2hphs\") pod \"thanos-querier-6db9fdb49c-5z6zr\" (UID: \"2cb59038-b544-4cb1-a2af-ded4bb5d6431\") " pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:43.579507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.579492 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x95q\" (UniqueName: \"kubernetes.io/projected/57df0df3-a039-45eb-8ae2-02433f6b0b2e-kube-api-access-8x95q\") pod \"network-metrics-daemon-xzjcg\" (UID: \"57df0df3-a039-45eb-8ae2-02433f6b0b2e\") " pod="openshift-multus/network-metrics-daemon-xzjcg" Oct 31 00:19:43.580874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.580857 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drh6b\" (UniqueName: \"kubernetes.io/projected/8a38f09d-12b6-481c-bb13-c686226854a6-kube-api-access-drh6b\") pod \"openshift-state-metrics-7bd7489957-mwvzf\" (UID: \"8a38f09d-12b6-481c-bb13-c686226854a6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-mwvzf" Oct 31 00:19:43.590300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.590279 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfq6z\" (UniqueName: \"kubernetes.io/projected/2b1af480-ae8f-483e-bd52-947094fcc3ff-kube-api-access-jfq6z\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.590583 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.590563 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwv2b\" (UniqueName: \"kubernetes.io/projected/465a3984-adbc-4677-b22f-9e9240a770f8-kube-api-access-rwv2b\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.591008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.590993 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khv2q\" (UniqueName: \"kubernetes.io/projected/abe218cc-f275-4474-a54e-3ba513e969c2-kube-api-access-khv2q\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.593631 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.593614 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5nl\" (UniqueName: \"kubernetes.io/projected/2d60169a-68c9-4cfd-bbd7-73ecac659695-kube-api-access-2v5nl\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.595537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.595521 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2pd\" (UniqueName: \"kubernetes.io/projected/364a8716-433d-4181-94ed-f6ccd33a0828-kube-api-access-6m2pd\") pod \"kube-state-metrics-5dc8d7d89b-mmhbf\" (UID: \"364a8716-433d-4181-94ed-f6ccd33a0828\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mmhbf" Oct 31 00:19:43.595996 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.595980 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kl9k\" (UniqueName: \"kubernetes.io/projected/d6f8e29e-fb86-4153-b168-0bb1743e44cd-kube-api-access-7kl9k\") pod \"dns-default-nqpmv\" (UID: \"d6f8e29e-fb86-4153-b168-0bb1743e44cd\") " pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:43.596542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.596523 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhtf\" (UniqueName: \"kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf\") pod \"console-fb49695f-qnzhd\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.596746 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.596730 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzq2\" (UniqueName: \"kubernetes.io/projected/05e75b3b-ffb1-4cda-9203-f4882bd385e7-kube-api-access-tfzq2\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.599075 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.599057 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnvmn\" (UniqueName: \"kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn\") pod \"console-5dcdc4b64f-p6xjm\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:19:43.602267 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.602249 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qh9\" (UniqueName: \"kubernetes.io/projected/f99f8918-3e3d-456b-be91-db4120b67424-kube-api-access-76qh9\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:19:43.607210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.607191 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4c5s\" (UniqueName: \"kubernetes.io/projected/0e0bd8aa-5627-4ece-8ef4-97a529ab89ae-kube-api-access-s4c5s\") pod \"metrics-server-6cbf58948-7xm9f\" (UID: \"0e0bd8aa-5627-4ece-8ef4-97a529ab89ae\") " pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:43.617362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.617337 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sk42\" (UniqueName: \"kubernetes.io/projected/b70404c4-e809-42c9-9efc-d6262e52815c-kube-api-access-2sk42\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:19:43.619211 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.619142 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb89\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89\") pod \"alertmanager-main-0\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:43.620102 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.620084 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzj5s\" (UniqueName: \"kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s\") pod \"console-69b4c98b6b-w7lch\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:43.621420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.621398 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwthq\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq\") pod \"prometheus-k8s-0\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:43.647179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647138 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47350ce3-ba5e-4393-ac0f-653b41bc28fd-hosts-file\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:19:43.647179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647175 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-cnibin\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.647419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647201 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-device-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.647419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647222 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-kubelet\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.647419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647238 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-hostroot\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.647419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647248 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47350ce3-ba5e-4393-ac0f-653b41bc28fd-hosts-file\") pod \"node-resolver-npncc\" (UID: \"47350ce3-ba5e-4393-ac0f-653b41bc28fd\") " pod="openshift-dns/node-resolver-npncc" Oct 31 00:19:43.647419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647313 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-device-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.647419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647326 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-kubelet\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.647419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647333 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-cnibin\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.647419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647378 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-hostroot\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.647419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647412 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-lib-modules\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647432 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647480 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-sys\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647497 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-run\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647508 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647512 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647539 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647548 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-lib-modules\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647553 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-conf\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647576 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-run\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647577 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-host\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647603 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-modprobe-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647616 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-host\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647625 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-multus-certs\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647580 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-sys\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647642 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysconfig\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647675 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-conf\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647676 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-k8s-cni-cncf-io\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647702 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-k8s-cni-cncf-io\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647714 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-socket-dir-parent\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647713 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-modprobe-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647716 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysconfig\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647733 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b70404c4-e809-42c9-9efc-d6262e52815c-host-slash\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:19:43.647726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647677 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-multus-certs\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647746 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-socket-dir-parent\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647753 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647763 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b70404c4-e809-42c9-9efc-d6262e52815c-host-slash\") pod \"iptables-alerter-b2slc\" (UID: \"b70404c4-e809-42c9-9efc-d6262e52815c\") " pod="openshift-network-operator/iptables-alerter-b2slc" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647774 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-registration-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647788 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647802 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-var-lib-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647812 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647826 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-system-cni-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647850 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-var-lib-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647857 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-slash\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647863 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d60169a-68c9-4cfd-bbd7-73ecac659695-system-cni-dir\") pod \"multus-additional-cni-plugins-mhppg\" (UID: \"2d60169a-68c9-4cfd-bbd7-73ecac659695\") " pod="openshift-multus/multus-additional-cni-plugins-mhppg" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647863 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-registration-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647882 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-var-lib-kubelet\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647891 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-sysctl-d\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647899 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-bin\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647914 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-etc-kubernetes\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647938 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647945 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-bin\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647963 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-slash\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647967 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-kubelet-config\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647973 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-var-lib-kubelet\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647990 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-dbus\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647999 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-kubelet-config\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.647963 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-etc-kubernetes\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648017 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648028 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648037 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a56c984b-c9d9-4d3f-86fd-2314f01a6673-dbus\") pod \"global-pull-secret-syncer-z4hfm\" (UID: \"a56c984b-c9d9-4d3f-86fd-2314f01a6673\") " pod="kube-system/global-pull-secret-syncer-z4hfm" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648066 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-etc-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648095 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-sys-fs\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648109 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648122 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-node-log\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648147 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abe218cc-f275-4474-a54e-3ba513e969c2-crio-socket\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648173 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648192 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-etc-openvswitch\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648201 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99f8918-3e3d-456b-be91-db4120b67424-host\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648176 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-sys-fs\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648199 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-node-log\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648211 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abe218cc-f275-4474-a54e-3ba513e969c2-crio-socket\") pod \"insights-runtime-extractor-qg2xf\" (UID: \"abe218cc-f275-4474-a54e-3ba513e969c2\") " pod="openshift-insights/insights-runtime-extractor-qg2xf" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648230 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99f8918-3e3d-456b-be91-db4120b67424-host\") pod \"node-ca-jpxt2\" (UID: \"f99f8918-3e3d-456b-be91-db4120b67424\") " pod="openshift-image-registry/node-ca-jpxt2" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648225 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648246 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-systemd\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648276 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-systemd\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648281 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-socket-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648301 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-conf-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648329 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-sys\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648338 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b1af480-ae8f-483e-bd52-947094fcc3ff-socket-dir\") pod \"aws-ebs-csi-driver-node-l6w9f\" (UID: \"2b1af480-ae8f-483e-bd52-947094fcc3ff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6w9f" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648352 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-bin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648374 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-multus-conf-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648380 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-netns\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648389 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-sys\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648408 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-ovn\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648413 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-bin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648424 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-run-netns\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648433 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-netd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648467 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-ovn\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648471 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-cnibin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648509 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-cni-netd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648513 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-cnibin\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648509 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-kubernetes\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648536 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-system-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648545 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e75b3b-ffb1-4cda-9203-f4882bd385e7-etc-kubernetes\") pod \"tuned-gffpv\" (UID: \"05e75b3b-ffb1-4cda-9203-f4882bd385e7\") " pod="openshift-cluster-node-tuning-operator/tuned-gffpv" Oct 31 00:19:43.648522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648557 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-wtmp\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648576 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-system-cni-dir\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648575 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-netns\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648594 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-run-netns\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648596 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-multus\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648617 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-cni-multus\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648623 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-log-socket\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648646 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-log-socket\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648653 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-node-exporter-wtmp\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648653 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-systemd-units\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648675 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-systemd-units\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648689 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-systemd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648712 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648737 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-run-systemd\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648767 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-kubelet\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648768 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/064705a9-54fd-4528-ae88-1cb4596c5b6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sgkkv\" (UID: \"064705a9-54fd-4528-ae88-1cb4596c5b6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648739 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af6b8f1-b029-4ae2-99cb-df0315daa268-host-var-lib-kubelet\") pod \"multus-bfsdv\" (UID: \"3af6b8f1-b029-4ae2-99cb-df0315daa268\") " pod="openshift-multus/multus-bfsdv" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648802 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-root\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.650218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.648853 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/465a3984-adbc-4677-b22f-9e9240a770f8-root\") pod \"node-exporter-lhrff\" (UID: \"465a3984-adbc-4677-b22f-9e9240a770f8\") " pod="openshift-monitoring/node-exporter-lhrff" Oct 31 00:19:43.923394 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.923309 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:43.928149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:43.928121 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:44.063179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:44.063148 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69b4c98b6b-w7lch"] Oct 31 00:19:44.250632 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:44.250556 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" Oct 31 00:19:44.573832 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:44.573796 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:44.593373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:44.593344 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:44.741821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:44.741788 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nqpmv" Oct 31 00:19:46.028308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:46.028278 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bshvl" Oct 31 00:19:47.819627 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:47.819591 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:47.968745 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:47.968708 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:49.002848 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:49.002821 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6cbf58948-7xm9f" Oct 31 00:19:49.536841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:49.536809 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:50.135895 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:50.135861 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-br9bn" Oct 31 00:19:50.693085 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:50.693050 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:50.745318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:50.745282 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:50.841506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:50.841475 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:51.315064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:51.315037 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:19:51.533179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:51.533150 5978 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:19:52.732806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:52.732774 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:19:52.984303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:52.984217 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:19:53.332499 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:53.332437 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:53.498350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:53.498311 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:54.649508 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:54.649448 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:19:54.649938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:54.649760 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="alertmanager" containerID="cri-o://2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" gracePeriod=120 Oct 31 00:19:54.649938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:54.649825 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy-metric" containerID="cri-o://b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" gracePeriod=120 Oct 31 00:19:54.649938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:54.649854 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="prom-label-proxy" containerID="cri-o://4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" gracePeriod=120 Oct 31 00:19:54.649938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:54.649890 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy-web" containerID="cri-o://716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" gracePeriod=120 Oct 31 00:19:54.650158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:54.649912 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="config-reloader" containerID="cri-o://d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" gracePeriod=120 Oct 31 00:19:54.650158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:54.649920 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy" containerID="cri-o://0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" gracePeriod=120 Oct 31 00:19:54.949199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:54.949162 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.064801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.064767 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.064969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.064810 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.064969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.064844 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.064969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.064888 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.064969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.064907 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.065162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065066 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.065162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065104 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.065162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065150 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.065319 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065188 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.065319 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065236 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.065319 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065283 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.065495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065306 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:55.065495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065317 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.065495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065402 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"90ede307-5582-4953-abaa-25d4393dc55c\" (UID: \"90ede307-5582-4953-abaa-25d4393dc55c\") " Oct 31 00:19:55.066050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065813 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:55.066050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065828 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-metrics-client-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.066050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.065863 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:55.069223 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.069193 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "kube-api-access-8mb89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:55.069342 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.069323 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:55.069411 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.069381 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:55.069744 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.069719 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:55.069847 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.069819 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:55.069953 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.069932 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:55.070035 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.070017 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:55.070644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.070615 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:55.074413 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.074386 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:55.081094 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.081070 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config" pod "90ede307-5582-4953-abaa-25d4393dc55c" (UID: "90ede307-5582-4953-abaa-25d4393dc55c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:55.166762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166681 5978 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-config-out\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.166762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166714 5978 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-main-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.166762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166725 5978 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.166762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166734 5978 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-web-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.166762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166743 5978 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-tls-assets\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.166762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166751 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mb89\" (UniqueName: \"kubernetes.io/projected/90ede307-5582-4953-abaa-25d4393dc55c-kube-api-access-8mb89\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.166762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166761 5978 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.166762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166770 5978 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.167114 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166779 5978 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-cluster-tls-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.167114 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166789 5978 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.167114 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166798 5978 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/90ede307-5582-4953-abaa-25d4393dc55c-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.167114 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.166806 5978 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/90ede307-5582-4953-abaa-25d4393dc55c-alertmanager-main-db\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:55.552305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552209 5978 generic.go:356] "Generic (PLEG): container finished" podID="90ede307-5582-4953-abaa-25d4393dc55c" containerID="4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" exitCode=0 Oct 31 00:19:55.552305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552241 5978 generic.go:356] "Generic (PLEG): container finished" podID="90ede307-5582-4953-abaa-25d4393dc55c" containerID="b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" exitCode=0 Oct 31 00:19:55.552305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552249 5978 generic.go:356] "Generic (PLEG): container finished" podID="90ede307-5582-4953-abaa-25d4393dc55c" containerID="0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" exitCode=0 Oct 31 00:19:55.552305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552257 5978 generic.go:356] "Generic (PLEG): container finished" podID="90ede307-5582-4953-abaa-25d4393dc55c" containerID="716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" exitCode=0 Oct 31 00:19:55.552305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552265 5978 generic.go:356] "Generic (PLEG): container finished" podID="90ede307-5582-4953-abaa-25d4393dc55c" containerID="d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" exitCode=0 Oct 31 00:19:55.552305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552275 5978 generic.go:356] "Generic (PLEG): container finished" podID="90ede307-5582-4953-abaa-25d4393dc55c" containerID="2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" exitCode=0 Oct 31 00:19:55.552305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552286 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerDied","Data":"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad"} Oct 31 00:19:55.552731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552311 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.552731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552362 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerDied","Data":"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7"} Oct 31 00:19:55.552731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552390 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerDied","Data":"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45"} Oct 31 00:19:55.552731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552416 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerDied","Data":"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8"} Oct 31 00:19:55.552731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552426 5978 scope.go:122] "RemoveContainer" containerID="4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" Oct 31 00:19:55.552731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552439 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerDied","Data":"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7"} Oct 31 00:19:55.552731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552484 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerDied","Data":"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7"} Oct 31 00:19:55.552731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.552512 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"90ede307-5582-4953-abaa-25d4393dc55c","Type":"ContainerDied","Data":"685c759650d3bc6423a5850f50eee487f651bd04752375115fc0ce094a86a517"} Oct 31 00:19:55.561392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.561372 5978 scope.go:122] "RemoveContainer" containerID="b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" Oct 31 00:19:55.569253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.569235 5978 scope.go:122] "RemoveContainer" containerID="0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" Oct 31 00:19:55.578650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.578625 5978 scope.go:122] "RemoveContainer" containerID="716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" Oct 31 00:19:55.585900 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.585878 5978 scope.go:122] "RemoveContainer" containerID="d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" Oct 31 00:19:55.593421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.593406 5978 scope.go:122] "RemoveContainer" containerID="2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" Oct 31 00:19:55.600649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.600629 5978 scope.go:122] "RemoveContainer" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" Oct 31 00:19:55.606056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.606035 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:19:55.607876 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.607859 5978 scope.go:122] "RemoveContainer" containerID="4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" Oct 31 00:19:55.608168 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:55.608145 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": container with ID starting with 4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad not found: ID does not exist" containerID="4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" Oct 31 00:19:55.608204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.608177 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad"} err="failed to get container status \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": rpc error: code = NotFound desc = could not find container \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": container with ID starting with 4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad not found: ID does not exist" Oct 31 00:19:55.608255 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.608207 5978 scope.go:122] "RemoveContainer" containerID="b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" Oct 31 00:19:55.608429 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:55.608413 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": container with ID starting with b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7 not found: ID does not exist" containerID="b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" Oct 31 00:19:55.608477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.608434 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7"} err="failed to get container status \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": rpc error: code = NotFound desc = could not find container \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": container with ID starting with b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7 not found: ID does not exist" Oct 31 00:19:55.608477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.608463 5978 scope.go:122] "RemoveContainer" containerID="0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" Oct 31 00:19:55.608659 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:55.608644 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": container with ID starting with 0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45 not found: ID does not exist" containerID="0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" Oct 31 00:19:55.608705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.608663 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45"} err="failed to get container status \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": rpc error: code = NotFound desc = could not find container \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": container with ID starting with 0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45 not found: ID does not exist" Oct 31 00:19:55.608705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.608675 5978 scope.go:122] "RemoveContainer" containerID="716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" Oct 31 00:19:55.608934 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:55.608915 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": container with ID starting with 716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8 not found: ID does not exist" containerID="716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" Oct 31 00:19:55.608970 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.608940 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8"} err="failed to get container status \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": rpc error: code = NotFound desc = could not find container \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": container with ID starting with 716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8 not found: ID does not exist" Oct 31 00:19:55.608970 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.608957 5978 scope.go:122] "RemoveContainer" containerID="d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" Oct 31 00:19:55.609187 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:55.609166 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": container with ID starting with d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7 not found: ID does not exist" containerID="d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" Oct 31 00:19:55.609260 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.609193 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7"} err="failed to get container status \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": rpc error: code = NotFound desc = could not find container \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": container with ID starting with d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7 not found: ID does not exist" Oct 31 00:19:55.609260 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.609211 5978 scope.go:122] "RemoveContainer" containerID="2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" Oct 31 00:19:55.609525 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:55.609463 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": container with ID starting with 2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7 not found: ID does not exist" containerID="2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" Oct 31 00:19:55.609525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.609494 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7"} err="failed to get container status \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": rpc error: code = NotFound desc = could not find container \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": container with ID starting with 2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7 not found: ID does not exist" Oct 31 00:19:55.609525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.609514 5978 scope.go:122] "RemoveContainer" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" Oct 31 00:19:55.609764 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:19:55.609745 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": container with ID starting with d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd not found: ID does not exist" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" Oct 31 00:19:55.609836 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.609772 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd"} err="failed to get container status \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": rpc error: code = NotFound desc = could not find container \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": container with ID starting with d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd not found: ID does not exist" Oct 31 00:19:55.609836 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.609792 5978 scope.go:122] "RemoveContainer" containerID="4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" Oct 31 00:19:55.610042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610025 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad"} err="failed to get container status \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": rpc error: code = NotFound desc = could not find container \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": container with ID starting with 4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad not found: ID does not exist" Oct 31 00:19:55.610089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610043 5978 scope.go:122] "RemoveContainer" containerID="b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" Oct 31 00:19:55.610253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610234 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7"} err="failed to get container status \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": rpc error: code = NotFound desc = could not find container \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": container with ID starting with b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7 not found: ID does not exist" Oct 31 00:19:55.610326 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610253 5978 scope.go:122] "RemoveContainer" containerID="0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" Oct 31 00:19:55.610485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610466 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45"} err="failed to get container status \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": rpc error: code = NotFound desc = could not find container \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": container with ID starting with 0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45 not found: ID does not exist" Oct 31 00:19:55.610543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610487 5978 scope.go:122] "RemoveContainer" containerID="716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" Oct 31 00:19:55.610707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610690 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8"} err="failed to get container status \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": rpc error: code = NotFound desc = could not find container \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": container with ID starting with 716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8 not found: ID does not exist" Oct 31 00:19:55.610753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610711 5978 scope.go:122] "RemoveContainer" containerID="d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" Oct 31 00:19:55.610908 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610891 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7"} err="failed to get container status \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": rpc error: code = NotFound desc = could not find container \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": container with ID starting with d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7 not found: ID does not exist" Oct 31 00:19:55.610974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.610909 5978 scope.go:122] "RemoveContainer" containerID="2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" Oct 31 00:19:55.611122 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.611106 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7"} err="failed to get container status \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": rpc error: code = NotFound desc = could not find container \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": container with ID starting with 2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7 not found: ID does not exist" Oct 31 00:19:55.611173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.611122 5978 scope.go:122] "RemoveContainer" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" Oct 31 00:19:55.611318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.611293 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd"} err="failed to get container status \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": rpc error: code = NotFound desc = could not find container \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": container with ID starting with d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd not found: ID does not exist" Oct 31 00:19:55.611394 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.611319 5978 scope.go:122] "RemoveContainer" containerID="4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" Oct 31 00:19:55.611540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.611524 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad"} err="failed to get container status \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": rpc error: code = NotFound desc = could not find container \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": container with ID starting with 4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad not found: ID does not exist" Oct 31 00:19:55.611540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.611540 5978 scope.go:122] "RemoveContainer" containerID="b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" Oct 31 00:19:55.611782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.611764 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7"} err="failed to get container status \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": rpc error: code = NotFound desc = could not find container \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": container with ID starting with b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7 not found: ID does not exist" Oct 31 00:19:55.611850 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.611783 5978 scope.go:122] "RemoveContainer" containerID="0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" Oct 31 00:19:55.616483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.611991 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45"} err="failed to get container status \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": rpc error: code = NotFound desc = could not find container \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": container with ID starting with 0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45 not found: ID does not exist" Oct 31 00:19:55.616483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.612343 5978 scope.go:122] "RemoveContainer" containerID="716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" Oct 31 00:19:55.616960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.616927 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8"} err="failed to get container status \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": rpc error: code = NotFound desc = could not find container \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": container with ID starting with 716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8 not found: ID does not exist" Oct 31 00:19:55.617044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.616960 5978 scope.go:122] "RemoveContainer" containerID="d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" Oct 31 00:19:55.617298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.617279 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7"} err="failed to get container status \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": rpc error: code = NotFound desc = could not find container \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": container with ID starting with d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7 not found: ID does not exist" Oct 31 00:19:55.617380 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.617302 5978 scope.go:122] "RemoveContainer" containerID="2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" Oct 31 00:19:55.617623 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.617588 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7"} err="failed to get container status \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": rpc error: code = NotFound desc = could not find container \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": container with ID starting with 2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7 not found: ID does not exist" Oct 31 00:19:55.617731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.617625 5978 scope.go:122] "RemoveContainer" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" Oct 31 00:19:55.618308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.617933 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd"} err="failed to get container status \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": rpc error: code = NotFound desc = could not find container \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": container with ID starting with d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd not found: ID does not exist" Oct 31 00:19:55.618308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.617956 5978 scope.go:122] "RemoveContainer" containerID="4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" Oct 31 00:19:55.618308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.618117 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:19:55.618308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.618271 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad"} err="failed to get container status \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": rpc error: code = NotFound desc = could not find container \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": container with ID starting with 4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad not found: ID does not exist" Oct 31 00:19:55.618308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.618289 5978 scope.go:122] "RemoveContainer" containerID="b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" Oct 31 00:19:55.618575 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.618525 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7"} err="failed to get container status \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": rpc error: code = NotFound desc = could not find container \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": container with ID starting with b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7 not found: ID does not exist" Oct 31 00:19:55.618575 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.618539 5978 scope.go:122] "RemoveContainer" containerID="0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" Oct 31 00:19:55.618757 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.618740 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45"} err="failed to get container status \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": rpc error: code = NotFound desc = could not find container \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": container with ID starting with 0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45 not found: ID does not exist" Oct 31 00:19:55.618817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.618757 5978 scope.go:122] "RemoveContainer" containerID="716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" Oct 31 00:19:55.618958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.618942 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8"} err="failed to get container status \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": rpc error: code = NotFound desc = could not find container \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": container with ID starting with 716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8 not found: ID does not exist" Oct 31 00:19:55.618958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.618956 5978 scope.go:122] "RemoveContainer" containerID="d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" Oct 31 00:19:55.619173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.619155 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7"} err="failed to get container status \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": rpc error: code = NotFound desc = could not find container \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": container with ID starting with d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7 not found: ID does not exist" Oct 31 00:19:55.619215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.619173 5978 scope.go:122] "RemoveContainer" containerID="2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" Oct 31 00:19:55.619385 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.619367 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7"} err="failed to get container status \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": rpc error: code = NotFound desc = could not find container \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": container with ID starting with 2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7 not found: ID does not exist" Oct 31 00:19:55.619427 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.619386 5978 scope.go:122] "RemoveContainer" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" Oct 31 00:19:55.619624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.619608 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd"} err="failed to get container status \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": rpc error: code = NotFound desc = could not find container \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": container with ID starting with d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd not found: ID does not exist" Oct 31 00:19:55.619624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.619623 5978 scope.go:122] "RemoveContainer" containerID="4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" Oct 31 00:19:55.619830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.619814 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad"} err="failed to get container status \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": rpc error: code = NotFound desc = could not find container \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": container with ID starting with 4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad not found: ID does not exist" Oct 31 00:19:55.619888 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.619830 5978 scope.go:122] "RemoveContainer" containerID="b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" Oct 31 00:19:55.620038 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620023 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7"} err="failed to get container status \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": rpc error: code = NotFound desc = could not find container \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": container with ID starting with b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7 not found: ID does not exist" Oct 31 00:19:55.620038 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620038 5978 scope.go:122] "RemoveContainer" containerID="0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" Oct 31 00:19:55.620248 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620231 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45"} err="failed to get container status \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": rpc error: code = NotFound desc = could not find container \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": container with ID starting with 0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45 not found: ID does not exist" Oct 31 00:19:55.620290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620248 5978 scope.go:122] "RemoveContainer" containerID="716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" Oct 31 00:19:55.620438 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620423 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8"} err="failed to get container status \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": rpc error: code = NotFound desc = could not find container \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": container with ID starting with 716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8 not found: ID does not exist" Oct 31 00:19:55.620501 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620438 5978 scope.go:122] "RemoveContainer" containerID="d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" Oct 31 00:19:55.620632 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620616 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7"} err="failed to get container status \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": rpc error: code = NotFound desc = could not find container \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": container with ID starting with d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7 not found: ID does not exist" Oct 31 00:19:55.620672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620631 5978 scope.go:122] "RemoveContainer" containerID="2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" Oct 31 00:19:55.620822 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620806 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7"} err="failed to get container status \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": rpc error: code = NotFound desc = could not find container \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": container with ID starting with 2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7 not found: ID does not exist" Oct 31 00:19:55.620858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620821 5978 scope.go:122] "RemoveContainer" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" Oct 31 00:19:55.620998 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620982 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd"} err="failed to get container status \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": rpc error: code = NotFound desc = could not find container \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": container with ID starting with d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd not found: ID does not exist" Oct 31 00:19:55.620998 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.620996 5978 scope.go:122] "RemoveContainer" containerID="4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad" Oct 31 00:19:55.621171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621156 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad"} err="failed to get container status \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": rpc error: code = NotFound desc = could not find container \"4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad\": container with ID starting with 4086801eaf8b60436c08cd8e6df4dbbe0b7351202b6f4f73eaa9f3242839bfad not found: ID does not exist" Oct 31 00:19:55.621214 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621171 5978 scope.go:122] "RemoveContainer" containerID="b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7" Oct 31 00:19:55.621360 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621343 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7"} err="failed to get container status \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": rpc error: code = NotFound desc = could not find container \"b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7\": container with ID starting with b5d6aeae8228c1d17346142ee2398d381e2748099c94e2fe803d08e25a1f77f7 not found: ID does not exist" Oct 31 00:19:55.621399 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621359 5978 scope.go:122] "RemoveContainer" containerID="0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45" Oct 31 00:19:55.621571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621556 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45"} err="failed to get container status \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": rpc error: code = NotFound desc = could not find container \"0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45\": container with ID starting with 0c09a78b5fdaf27be3c71a9fbd51e821a34cd32efae2defa519cd80f26c8ae45 not found: ID does not exist" Oct 31 00:19:55.621571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621570 5978 scope.go:122] "RemoveContainer" containerID="716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8" Oct 31 00:19:55.621758 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621743 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8"} err="failed to get container status \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": rpc error: code = NotFound desc = could not find container \"716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8\": container with ID starting with 716e3fdd433d88a38d3002a0880031c55f264fc5d7cb851efd12dc881de3c6d8 not found: ID does not exist" Oct 31 00:19:55.621801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621757 5978 scope.go:122] "RemoveContainer" containerID="d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7" Oct 31 00:19:55.621969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621954 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7"} err="failed to get container status \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": rpc error: code = NotFound desc = could not find container \"d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7\": container with ID starting with d6bfd84ef480ea08185f9717b3ede7b1958a2804d2f9b6a79ff753a2398c3ca7 not found: ID does not exist" Oct 31 00:19:55.621969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.621967 5978 scope.go:122] "RemoveContainer" containerID="2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7" Oct 31 00:19:55.622150 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.622134 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7"} err="failed to get container status \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": rpc error: code = NotFound desc = could not find container \"2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7\": container with ID starting with 2b97ba3fe2efd54f5cdae8bfa43e03769c1126a899243c5b8a0273b4d1a578d7 not found: ID does not exist" Oct 31 00:19:55.622193 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.622150 5978 scope.go:122] "RemoveContainer" containerID="d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd" Oct 31 00:19:55.622350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.622332 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd"} err="failed to get container status \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": rpc error: code = NotFound desc = could not find container \"d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd\": container with ID starting with d982ab16fc4133c83f2e9cf413bd7b3d215b0ad721636f1352278b6352755abd not found: ID does not exist" Oct 31 00:19:55.644580 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.644543 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:19:55.645046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645025 5978 cpu_manager.go:414] "containerMap: removing container" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy-metric" Oct 31 00:19:55.645046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645044 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy-metric" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645056 5978 cpu_manager.go:414] "containerMap: removing container" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="init-config-reloader" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645062 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="init-config-reloader" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645072 5978 cpu_manager.go:414] "containerMap: removing container" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="config-reloader" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645077 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="config-reloader" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645089 5978 cpu_manager.go:414] "containerMap: removing container" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="prom-label-proxy" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645094 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="prom-label-proxy" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645099 5978 cpu_manager.go:414] "containerMap: removing container" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645104 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645110 5978 cpu_manager.go:414] "containerMap: removing container" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="alertmanager" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645115 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="alertmanager" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645127 5978 cpu_manager.go:414] "containerMap: removing container" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy-web" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645131 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy-web" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645180 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy-metric" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645187 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="init-config-reloader" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645194 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="config-reloader" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645200 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="prom-label-proxy" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645205 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645210 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="alertmanager" Oct 31 00:19:55.645221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.645216 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ede307-5582-4953-abaa-25d4393dc55c" containerName="kube-rbac-proxy-web" Oct 31 00:19:55.677644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.677612 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.678606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.678522 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:19:55.693330 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.693301 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:19:55.693486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.693359 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:19:55.693585 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.693571 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:19:55.693787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.693749 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:19:55.693787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.693754 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:19:55.693950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.693789 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:19:55.693950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.693782 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:19:55.693950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.693853 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:19:55.693950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.693856 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vhm6s\"" Oct 31 00:19:55.705632 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.705610 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:19:55.873199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873162 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-web-config\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873198 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873223 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873365 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873417 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873497 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873549 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873585 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873638 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873605 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjswj\" (UniqueName: \"kubernetes.io/projected/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-kube-api-access-zjswj\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873658 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-config-out\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873717 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873702 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873758 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873747 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-config-volume\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.873804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.873791 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-tls-assets\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975251 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975202 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975263 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975287 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjswj\" (UniqueName: \"kubernetes.io/projected/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-kube-api-access-zjswj\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975316 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-config-out\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975347 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975377 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-config-volume\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975400 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-tls-assets\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975482 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-web-config\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975511 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975543 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975575 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975625 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975653 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.975787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975766 5978 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:19:55.976482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.976362 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.976482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.975770 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.976639 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.976489 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.978404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.978376 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.978676 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.978618 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-config-out\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.978793 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.978704 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.978793 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.978776 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.978896 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.978839 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.979118 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.979102 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.979216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.979197 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-config-volume\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.979870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.979853 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-tls-assets\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.980520 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.980505 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-web-config\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:55.998856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:55.998819 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjswj\" (UniqueName: \"kubernetes.io/projected/99e45dfd-1c7d-4ca5-92ae-65e99eef0e30-kube-api-access-zjswj\") pod \"alertmanager-main-0\" (UID: \"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:56.064258 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:56.064228 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:19:56.424353 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:56.424316 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ede307-5582-4953-abaa-25d4393dc55c" path="/var/lib/kubelet/pods/90ede307-5582-4953-abaa-25d4393dc55c/volumes" Oct 31 00:19:56.424499 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:19:56.424371 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e45dfd_1c7d_4ca5_92ae_65e99eef0e30.slice/crio-6e3a3b13d55f0cf606edc656263d566e6c06dd36ad946800eac1ec9b7ec2d446 WatchSource:0}: Error finding container 6e3a3b13d55f0cf606edc656263d566e6c06dd36ad946800eac1ec9b7ec2d446: Status 404 returned error can't find the container with id 6e3a3b13d55f0cf606edc656263d566e6c06dd36ad946800eac1ec9b7ec2d446 Oct 31 00:19:56.424964 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:56.424942 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:19:56.525012 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:56.524986 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:56.557077 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:56.557044 5978 generic.go:356] "Generic (PLEG): container finished" podID="99e45dfd-1c7d-4ca5-92ae-65e99eef0e30" containerID="ed312625b517b78f0284c5cd235c1b4ca46ad3e29cdb9e4b330528a817d7d524" exitCode=0 Oct 31 00:19:56.557225 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:56.557124 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30","Type":"ContainerDied","Data":"ed312625b517b78f0284c5cd235c1b4ca46ad3e29cdb9e4b330528a817d7d524"} Oct 31 00:19:56.557225 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:56.557193 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30","Type":"ContainerStarted","Data":"6e3a3b13d55f0cf606edc656263d566e6c06dd36ad946800eac1ec9b7ec2d446"} Oct 31 00:19:57.565707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:57.565669 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30","Type":"ContainerStarted","Data":"512473165a2c081b862e67f61d9a67c41a858f14fad77ee6733c2fd0849cf85d"} Oct 31 00:19:57.566158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:57.565785 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30","Type":"ContainerStarted","Data":"3d4ca2d032f08e01e2c778ee04376340a732558864123093b0ef9a922b87f10c"} Oct 31 00:19:57.566158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:57.565832 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30","Type":"ContainerStarted","Data":"de827bc09d991c0792fba60a9c907c3a98f5aba521184662c6079ed9abe3d00c"} Oct 31 00:19:57.566158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:57.565870 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30","Type":"ContainerStarted","Data":"52b385297181f18ae802d9304fa2ecdf1792a4d2fbad047a55ee8fd8b568a2c2"} Oct 31 00:19:57.566158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:57.565907 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30","Type":"ContainerStarted","Data":"9736d6f98c2de322c2d9c99f478b5eeaa96c66275e69407178f152cbe071d077"} Oct 31 00:19:57.566158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:57.565940 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99e45dfd-1c7d-4ca5-92ae-65e99eef0e30","Type":"ContainerStarted","Data":"6af03c744cb7c14239e19644cf6efb18056f9231d08872962a834d665b4ee1d8"} Oct 31 00:19:57.631719 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:57.631659 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.607360682 podStartE2EDuration="2.631646142s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.28546ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:57.630894823 +0000 UTC m=+15.329008625" watchObservedRunningTime="2026-10-31 00:19:57.631646142 +0000 UTC m=+15.329759898" Oct 31 00:19:59.058513 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.058466 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:59.334599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.334563 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:19:59.334937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.334900 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="prometheus" containerID="cri-o://6aba1a03018f85efb484fdd0764a442f278064adbce382d60d4e23c945be49fe" gracePeriod=600 Oct 31 00:19:59.335286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.335258 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ac271606873784c21badc01029a2e1a3bb847c9ad7ab3bdf28725a4a083c3010" gracePeriod=600 Oct 31 00:19:59.335379 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.335359 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy" containerID="cri-o://1bf03022442047c5405a49f4b0aa49d12fcf1189aff9b1151a9890ee13d5b599" gracePeriod=600 Oct 31 00:19:59.335431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.335407 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy-web" containerID="cri-o://01c06835981e0b68637cfca8ee4f2521cb0f12ecf90b85e80368126a98e67299" gracePeriod=600 Oct 31 00:19:59.335495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.335479 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="thanos-sidecar" containerID="cri-o://6b2f498915bfa2be8ed136ce18d689919db669e615ceceb13314838421a5a9bb" gracePeriod=600 Oct 31 00:19:59.335612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.335557 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="config-reloader" containerID="cri-o://8c441888f2d6f0588f264a3c462f47388be96dcb0127c6c7858dce4078962ded" gracePeriod=600 Oct 31 00:19:59.542298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.542270 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:19:59.578756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578722 5978 generic.go:356] "Generic (PLEG): container finished" podID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerID="ac271606873784c21badc01029a2e1a3bb847c9ad7ab3bdf28725a4a083c3010" exitCode=0 Oct 31 00:19:59.578756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578756 5978 generic.go:356] "Generic (PLEG): container finished" podID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerID="1bf03022442047c5405a49f4b0aa49d12fcf1189aff9b1151a9890ee13d5b599" exitCode=0 Oct 31 00:19:59.578950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578767 5978 generic.go:356] "Generic (PLEG): container finished" podID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerID="01c06835981e0b68637cfca8ee4f2521cb0f12ecf90b85e80368126a98e67299" exitCode=0 Oct 31 00:19:59.578950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578779 5978 generic.go:356] "Generic (PLEG): container finished" podID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerID="6b2f498915bfa2be8ed136ce18d689919db669e615ceceb13314838421a5a9bb" exitCode=0 Oct 31 00:19:59.578950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578787 5978 generic.go:356] "Generic (PLEG): container finished" podID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerID="8c441888f2d6f0588f264a3c462f47388be96dcb0127c6c7858dce4078962ded" exitCode=0 Oct 31 00:19:59.578950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578791 5978 generic.go:356] "Generic (PLEG): container finished" podID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerID="6aba1a03018f85efb484fdd0764a442f278064adbce382d60d4e23c945be49fe" exitCode=0 Oct 31 00:19:59.578950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578797 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerDied","Data":"ac271606873784c21badc01029a2e1a3bb847c9ad7ab3bdf28725a4a083c3010"} Oct 31 00:19:59.578950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578867 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerDied","Data":"1bf03022442047c5405a49f4b0aa49d12fcf1189aff9b1151a9890ee13d5b599"} Oct 31 00:19:59.578950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578907 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerDied","Data":"01c06835981e0b68637cfca8ee4f2521cb0f12ecf90b85e80368126a98e67299"} Oct 31 00:19:59.578950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578936 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerDied","Data":"6b2f498915bfa2be8ed136ce18d689919db669e615ceceb13314838421a5a9bb"} Oct 31 00:19:59.579191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578966 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerDied","Data":"8c441888f2d6f0588f264a3c462f47388be96dcb0127c6c7858dce4078962ded"} Oct 31 00:19:59.579191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.578991 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerDied","Data":"6aba1a03018f85efb484fdd0764a442f278064adbce382d60d4e23c945be49fe"} Oct 31 00:19:59.594282 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.594227 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:19:59.601817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.601798 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.601949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.601832 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.601949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.601871 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602078 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602060 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602099 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602189 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602133 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602189 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602160 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602189 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602272 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:59.602515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602444 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602591 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602517 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:59.602591 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602556 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602599 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602632 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602664 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602694 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602711 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:59.602838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602726 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602728 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:59.602838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602773 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602803 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.602838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602836 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.603191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602879 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.603191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.602925 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"13ec7561-c654-4156-9ec3-5cb5893881b8\" (UID: \"13ec7561-c654-4156-9ec3-5cb5893881b8\") " Oct 31 00:19:59.603191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.603126 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:59.603357 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.603335 5978 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.603408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.603363 5978 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.603408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.603381 5978 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.603408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.603396 5978 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-configmap-metrics-client-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.603599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.603410 5978 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.603651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.603618 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:59.603893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.603835 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:59.605162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.605133 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:59.605316 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.605291 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.605542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.605501 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.605749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.605636 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.606097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.606071 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.606621 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.606590 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "kube-api-access-kwthq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:59.606737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.606708 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.607339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.607319 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:59.607422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.607340 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:59.607603 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.607583 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.607809 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.607788 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.608192 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.608176 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.616977 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.616950 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config" pod "13ec7561-c654-4156-9ec3-5cb5893881b8" (UID: "13ec7561-c654-4156-9ec3-5cb5893881b8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:59.703789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703756 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwthq\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-kube-api-access-kwthq\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.703789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703784 5978 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.703789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703794 5978 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-metrics-client-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703803 5978 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703812 5978 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703823 5978 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703831 5978 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13ec7561-c654-4156-9ec3-5cb5893881b8-tls-assets\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703840 5978 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-web-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703847 5978 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703855 5978 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-config-out\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703863 5978 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-kube-rbac-proxy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703871 5978 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703880 5978 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-grpc-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703888 5978 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13ec7561-c654-4156-9ec3-5cb5893881b8-prometheus-k8s-db\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:19:59.704000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:19:59.703896 5978 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13ec7561-c654-4156-9ec3-5cb5893881b8-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:00.060865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.060762 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:00.257317 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.257275 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:00.586360 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.586327 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13ec7561-c654-4156-9ec3-5cb5893881b8","Type":"ContainerDied","Data":"5a446832b1b0a1c34d8f267e9f78e5eef1e4b8b3e25f0ce756fbf78403ed4593"} Oct 31 00:20:00.586557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.586383 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.586557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.586413 5978 scope.go:122] "RemoveContainer" containerID="ac271606873784c21badc01029a2e1a3bb847c9ad7ab3bdf28725a4a083c3010" Oct 31 00:20:00.594548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.594528 5978 scope.go:122] "RemoveContainer" containerID="1bf03022442047c5405a49f4b0aa49d12fcf1189aff9b1151a9890ee13d5b599" Oct 31 00:20:00.601793 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.601770 5978 scope.go:122] "RemoveContainer" containerID="01c06835981e0b68637cfca8ee4f2521cb0f12ecf90b85e80368126a98e67299" Oct 31 00:20:00.608925 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.608904 5978 scope.go:122] "RemoveContainer" containerID="6b2f498915bfa2be8ed136ce18d689919db669e615ceceb13314838421a5a9bb" Oct 31 00:20:00.618377 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.618353 5978 scope.go:122] "RemoveContainer" containerID="8c441888f2d6f0588f264a3c462f47388be96dcb0127c6c7858dce4078962ded" Oct 31 00:20:00.625763 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.625688 5978 scope.go:122] "RemoveContainer" containerID="6aba1a03018f85efb484fdd0764a442f278064adbce382d60d4e23c945be49fe" Oct 31 00:20:00.633620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.633599 5978 scope.go:122] "RemoveContainer" containerID="f94e4d0b68df9df554ee8f6a4eec3a65d85d4d43ac8df05dd72a3b0c381191cc" Oct 31 00:20:00.655567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.655540 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:20:00.667691 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.667666 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:20:00.703178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703144 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:20:00.703480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703446 5978 cpu_manager.go:414] "containerMap: removing container" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="init-config-reloader" Oct 31 00:20:00.703523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703477 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="init-config-reloader" Oct 31 00:20:00.703523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703497 5978 cpu_manager.go:414] "containerMap: removing container" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="config-reloader" Oct 31 00:20:00.703523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703502 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="config-reloader" Oct 31 00:20:00.703523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703513 5978 cpu_manager.go:414] "containerMap: removing container" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy-thanos" Oct 31 00:20:00.703523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703519 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy-thanos" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703526 5978 cpu_manager.go:414] "containerMap: removing container" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703532 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703544 5978 cpu_manager.go:414] "containerMap: removing container" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy-web" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703549 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy-web" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703556 5978 cpu_manager.go:414] "containerMap: removing container" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="thanos-sidecar" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703560 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="thanos-sidecar" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703576 5978 cpu_manager.go:414] "containerMap: removing container" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="prometheus" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703580 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="prometheus" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703629 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy-web" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703636 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="thanos-sidecar" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703646 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="prometheus" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703652 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="init-config-reloader" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703657 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="config-reloader" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703663 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy-thanos" Oct 31 00:20:00.703683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.703667 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" containerName="kube-rbac-proxy" Oct 31 00:20:00.707522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.707504 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.711776 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.711746 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.711901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.711790 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-web-config\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.711901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.711891 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/011c406e-96bb-49b6-bdd8-d4f2b36d636e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.711972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.711921 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-config\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.711972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.711948 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712037 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712024 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712071 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712060 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712123 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712151 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712230 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712205 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712296 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712375 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712324 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712375 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712357 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712407 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712480 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712512 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712536 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjdj\" (UniqueName: \"kubernetes.io/projected/011c406e-96bb-49b6-bdd8-d4f2b36d636e-kube-api-access-jzjdj\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712563 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712681 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712667 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.712732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.712703 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/011c406e-96bb-49b6-bdd8-d4f2b36d636e-config-out\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.724942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.724918 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:20:00.725204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725187 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:20:00.725523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725506 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:20:00.725584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725520 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:20:00.725584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725542 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:20:00.725584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725555 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:20:00.725683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725613 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:20:00.725683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725669 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:20:00.725742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725695 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:20:00.725742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725707 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:20:00.725742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725700 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:20:00.725742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725732 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mg4tg\"" Oct 31 00:20:00.725923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.725754 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cno706g3ng8gr\"" Oct 31 00:20:00.726054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.726036 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:20:00.738920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.738886 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:20:00.739256 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.739235 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:20:00.742032 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.742007 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:20:00.814069 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814033 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814069 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814072 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814093 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814117 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814145 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814254 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814284 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjdj\" (UniqueName: \"kubernetes.io/projected/011c406e-96bb-49b6-bdd8-d4f2b36d636e-kube-api-access-jzjdj\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814309 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814338 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814370 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/011c406e-96bb-49b6-bdd8-d4f2b36d636e-config-out\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814419 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814442 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-web-config\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814506 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/011c406e-96bb-49b6-bdd8-d4f2b36d636e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814553 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-config\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814588 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814611 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814638 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814667 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814689 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.814738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814719 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.815142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.814830 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.815293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.815271 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.815946 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.815921 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.816114 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.816088 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.817265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.816604 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.817370 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.817331 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.817532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.817504 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.817644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.817512 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.817708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.817678 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.817805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.817781 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.818617 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.818553 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.819032 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.818990 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.819512 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.819434 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/011c406e-96bb-49b6-bdd8-d4f2b36d636e-config-out\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.819887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.819867 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.820035 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.820014 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/011c406e-96bb-49b6-bdd8-d4f2b36d636e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.820100 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.820083 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.820495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.820478 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-config\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.820579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.820527 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/011c406e-96bb-49b6-bdd8-d4f2b36d636e-web-config\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.820834 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.820814 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/011c406e-96bb-49b6-bdd8-d4f2b36d636e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:00.838496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:00.838469 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjdj\" (UniqueName: \"kubernetes.io/projected/011c406e-96bb-49b6-bdd8-d4f2b36d636e-kube-api-access-jzjdj\") pod \"prometheus-k8s-0\" (UID: \"011c406e-96bb-49b6-bdd8-d4f2b36d636e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:01.025609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:01.025518 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:01.361490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:01.361445 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:20:01.362299 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:20:01.362272 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011c406e_96bb_49b6_bdd8_d4f2b36d636e.slice/crio-c034ad4d784511f0da6ee208ff9a2fe14f3e6bfa78643a445a3ddd608d33579b WatchSource:0}: Error finding container c034ad4d784511f0da6ee208ff9a2fe14f3e6bfa78643a445a3ddd608d33579b: Status 404 returned error can't find the container with id c034ad4d784511f0da6ee208ff9a2fe14f3e6bfa78643a445a3ddd608d33579b Oct 31 00:20:01.591572 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:01.591534 5978 generic.go:356] "Generic (PLEG): container finished" podID="011c406e-96bb-49b6-bdd8-d4f2b36d636e" containerID="efb996d3600a92246b7890f96501686dd23606231e8e85eb098a77ec7b15a6eb" exitCode=0 Oct 31 00:20:01.591769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:01.591614 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"011c406e-96bb-49b6-bdd8-d4f2b36d636e","Type":"ContainerDied","Data":"efb996d3600a92246b7890f96501686dd23606231e8e85eb098a77ec7b15a6eb"} Oct 31 00:20:01.591769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:01.591697 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"011c406e-96bb-49b6-bdd8-d4f2b36d636e","Type":"ContainerStarted","Data":"c034ad4d784511f0da6ee208ff9a2fe14f3e6bfa78643a445a3ddd608d33579b"} Oct 31 00:20:02.424053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:02.424020 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ec7561-c654-4156-9ec3-5cb5893881b8" path="/var/lib/kubelet/pods/13ec7561-c654-4156-9ec3-5cb5893881b8/volumes" Oct 31 00:20:02.603652 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:02.603617 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"011c406e-96bb-49b6-bdd8-d4f2b36d636e","Type":"ContainerStarted","Data":"7f0e585a9d29b4ff22645f3c947fa91671a44dcb6047c503bfc0a799f1ee6e54"} Oct 31 00:20:02.603792 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:02.603712 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"011c406e-96bb-49b6-bdd8-d4f2b36d636e","Type":"ContainerStarted","Data":"60264e91642a5e32f4468ff5b46fcb57a19c20efadb869c1ef7b1e6f09264eb8"} Oct 31 00:20:02.603792 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:02.603742 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"011c406e-96bb-49b6-bdd8-d4f2b36d636e","Type":"ContainerStarted","Data":"02f737495c37d36eec9f373e024a7cd7a9aa49ee0f0e2fadb92045013cdae9d2"} Oct 31 00:20:02.603792 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:02.603765 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"011c406e-96bb-49b6-bdd8-d4f2b36d636e","Type":"ContainerStarted","Data":"9aabfbcd5d10040a7c4039322be155ff668a8ecfd4b5dbdf7b0fff062da23784"} Oct 31 00:20:02.603906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:02.603805 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"011c406e-96bb-49b6-bdd8-d4f2b36d636e","Type":"ContainerStarted","Data":"5c571a3119de2cd8aeb8865ff32032ab01af114d3e5c6f38d212eda3b0973486"} Oct 31 00:20:02.603906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:02.603834 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"011c406e-96bb-49b6-bdd8-d4f2b36d636e","Type":"ContainerStarted","Data":"798957f542294ce5e4c1c3f8401ee3bcda90bd00a2a10c031a46d4b02b3691d6"} Oct 31 00:20:02.682209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:02.682107 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.664817554 podStartE2EDuration="2.682094493s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.276939ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:02.680307784 +0000 UTC m=+20.378421549" watchObservedRunningTime="2026-10-31 00:20:02.682094493 +0000 UTC m=+20.380208257" Oct 31 00:20:02.811748 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:02.811717 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:03.014648 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:03.014563 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:03.179928 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:20:03.179888 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:20:03.192358 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:03.192329 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kqx7w_7dd2bec9-6796-4e26-ad19-bfd15842852b/serve-healthcheck-canary/0.log" Oct 31 00:20:04.455020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.454989 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-68467f5bb4-tgfc8"] Oct 31 00:20:04.457939 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.457920 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.485536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.485504 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68467f5bb4-tgfc8"] Oct 31 00:20:04.552591 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.552560 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-oauth-config\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.552761 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.552612 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-trusted-ca-bundle\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.552761 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.552689 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-service-ca\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.552761 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.552727 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgwn\" (UniqueName: \"kubernetes.io/projected/54afbbc8-2da3-4771-9adf-917639334382-kube-api-access-tvgwn\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.552870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.552806 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-serving-cert\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.552870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.552835 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-console-config\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.552870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.552858 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-oauth-serving-cert\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.552978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.552931 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54afbbc8-2da3-4771-9adf-917639334382-tmp\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.615752 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.615711 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:04.653965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.653930 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-serving-cert\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.653965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.653976 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-console-config\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.654216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.654037 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-oauth-serving-cert\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.654216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.654089 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54afbbc8-2da3-4771-9adf-917639334382-tmp\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.654216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.654114 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-oauth-config\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.654216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.654168 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-trusted-ca-bundle\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.654216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.654195 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-service-ca\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.654500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.654221 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgwn\" (UniqueName: \"kubernetes.io/projected/54afbbc8-2da3-4771-9adf-917639334382-kube-api-access-tvgwn\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.655007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.654979 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-oauth-serving-cert\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.655137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.655059 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-console-config\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.655206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.655137 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-trusted-ca-bundle\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.655267 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.655245 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54afbbc8-2da3-4771-9adf-917639334382-tmp\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.655504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.655486 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-service-ca\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.656701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.656677 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-serving-cert\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.656812 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.656737 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-oauth-config\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.679487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.679437 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgwn\" (UniqueName: \"kubernetes.io/projected/54afbbc8-2da3-4771-9adf-917639334382-kube-api-access-tvgwn\") pod \"console-68467f5bb4-tgfc8\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.768427 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.768338 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:04.768427 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.768397 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:04.921387 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:04.921329 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:05.067342 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:05.067317 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68467f5bb4-tgfc8"] Oct 31 00:20:05.068520 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:05.068485 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:05.069266 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:20:05.069244 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54afbbc8_2da3_4771_9adf_917639334382.slice/crio-60e0e2378ac0eedcf46f297d4e16fffbefcf0ff25b9d73acb09b2a4e7f430b10 WatchSource:0}: Error finding container 60e0e2378ac0eedcf46f297d4e16fffbefcf0ff25b9d73acb09b2a4e7f430b10: Status 404 returned error can't find the container with id 60e0e2378ac0eedcf46f297d4e16fffbefcf0ff25b9d73acb09b2a4e7f430b10 Oct 31 00:20:05.618056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:05.618021 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68467f5bb4-tgfc8" event={"ID":"54afbbc8-2da3-4771-9adf-917639334382","Type":"ContainerStarted","Data":"08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d"} Oct 31 00:20:05.618504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:05.618076 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68467f5bb4-tgfc8" event={"ID":"54afbbc8-2da3-4771-9adf-917639334382","Type":"ContainerStarted","Data":"60e0e2378ac0eedcf46f297d4e16fffbefcf0ff25b9d73acb09b2a4e7f430b10"} Oct 31 00:20:05.664720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:05.662775 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-68467f5bb4-tgfc8" podStartSLOduration=1.662755535 podStartE2EDuration="1.662755535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:05.661542067 +0000 UTC m=+23.359655833" watchObservedRunningTime="2026-10-31 00:20:05.662755535 +0000 UTC m=+23.360869305" Oct 31 00:20:06.026120 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:06.026027 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:08.438819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.438785 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5dcdc4b64f-p6xjm" podUID="0fa8cd46-8095-4e93-b23a-0a422a47eab3" containerName="console" containerID="cri-o://e975cdcf919ecf22befb787d8d84d9f8d259de811c42246a389919bcac9e5531" gracePeriod=15 Oct 31 00:20:08.632890 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.632849 5978 generic.go:356] "Generic (PLEG): container finished" podID="0fa8cd46-8095-4e93-b23a-0a422a47eab3" containerID="e975cdcf919ecf22befb787d8d84d9f8d259de811c42246a389919bcac9e5531" exitCode=0 Oct 31 00:20:08.633069 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.632912 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcdc4b64f-p6xjm" event={"ID":"0fa8cd46-8095-4e93-b23a-0a422a47eab3","Type":"ContainerDied","Data":"e975cdcf919ecf22befb787d8d84d9f8d259de811c42246a389919bcac9e5531"} Oct 31 00:20:08.690515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.690436 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:20:08.795862 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.795824 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca\") pod \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " Oct 31 00:20:08.796063 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.795879 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config\") pod \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " Oct 31 00:20:08.796063 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.795915 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp\") pod \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " Oct 31 00:20:08.796063 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.795955 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config\") pod \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " Oct 31 00:20:08.796063 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796049 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert\") pod \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " Oct 31 00:20:08.796292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796241 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert\") pod \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " Oct 31 00:20:08.796351 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796280 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp" pod "0fa8cd46-8095-4e93-b23a-0a422a47eab3" (UID: "0fa8cd46-8095-4e93-b23a-0a422a47eab3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:08.796351 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796336 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn\" (UniqueName: \"kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn\") pod \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\" (UID: \"0fa8cd46-8095-4e93-b23a-0a422a47eab3\") " Oct 31 00:20:08.796479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796393 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca" pod "0fa8cd46-8095-4e93-b23a-0a422a47eab3" (UID: "0fa8cd46-8095-4e93-b23a-0a422a47eab3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:08.796479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796408 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config" pod "0fa8cd46-8095-4e93-b23a-0a422a47eab3" (UID: "0fa8cd46-8095-4e93-b23a-0a422a47eab3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:08.796598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796563 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert" pod "0fa8cd46-8095-4e93-b23a-0a422a47eab3" (UID: "0fa8cd46-8095-4e93-b23a-0a422a47eab3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:08.796639 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796620 5978 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:08.796639 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796634 5978 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:08.796701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796643 5978 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa8cd46-8095-4e93-b23a-0a422a47eab3-tmp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:08.796701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.796652 5978 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fa8cd46-8095-4e93-b23a-0a422a47eab3-oauth-serving-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:08.798412 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.798386 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert" pod "0fa8cd46-8095-4e93-b23a-0a422a47eab3" (UID: "0fa8cd46-8095-4e93-b23a-0a422a47eab3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:08.798565 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.798427 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn" pod "0fa8cd46-8095-4e93-b23a-0a422a47eab3" (UID: "0fa8cd46-8095-4e93-b23a-0a422a47eab3"). InnerVolumeSpecName "kube-api-access-pnvmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:08.798565 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.798465 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config" pod "0fa8cd46-8095-4e93-b23a-0a422a47eab3" (UID: "0fa8cd46-8095-4e93-b23a-0a422a47eab3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:08.897304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.897249 5978 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-serving-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:08.897304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.897299 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnvmn\" (UniqueName: \"kubernetes.io/projected/0fa8cd46-8095-4e93-b23a-0a422a47eab3-kube-api-access-pnvmn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:08.897304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.897313 5978 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fa8cd46-8095-4e93-b23a-0a422a47eab3-console-oauth-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:08.935522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:08.935494 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sgkkv" Oct 31 00:20:09.522616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.522575 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-69b4c98b6b-w7lch" podUID="0dae54b3-dceb-4795-85ec-42f9fb9ff256" containerName="console" containerID="cri-o://7dba283abe3a374371a9fe2b35905e6b1cca9b421835f0bddb66506b02d20dbc" gracePeriod=15 Oct 31 00:20:09.639223 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.639186 5978 generic.go:356] "Generic (PLEG): container finished" podID="0dae54b3-dceb-4795-85ec-42f9fb9ff256" containerID="7dba283abe3a374371a9fe2b35905e6b1cca9b421835f0bddb66506b02d20dbc" exitCode=0 Oct 31 00:20:09.639378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.639260 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b4c98b6b-w7lch" event={"ID":"0dae54b3-dceb-4795-85ec-42f9fb9ff256","Type":"ContainerDied","Data":"7dba283abe3a374371a9fe2b35905e6b1cca9b421835f0bddb66506b02d20dbc"} Oct 31 00:20:09.640694 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.640668 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcdc4b64f-p6xjm" event={"ID":"0fa8cd46-8095-4e93-b23a-0a422a47eab3","Type":"ContainerDied","Data":"606f1de1f43b9934f13b8fe518dffe1d71692f202bbbc2d49e76ef0e988e8bbf"} Oct 31 00:20:09.640800 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.640720 5978 scope.go:122] "RemoveContainer" containerID="e975cdcf919ecf22befb787d8d84d9f8d259de811c42246a389919bcac9e5531" Oct 31 00:20:09.640800 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.640721 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcdc4b64f-p6xjm" Oct 31 00:20:09.690292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.690259 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dcdc4b64f-p6xjm"] Oct 31 00:20:09.702559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.702521 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dcdc4b64f-p6xjm"] Oct 31 00:20:09.772063 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.772036 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:20:09.806208 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806128 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert\") pod \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " Oct 31 00:20:09.806208 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806171 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp\") pod \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " Oct 31 00:20:09.806380 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806217 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert\") pod \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " Oct 31 00:20:09.806380 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806332 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle\") pod \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " Oct 31 00:20:09.806380 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806373 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca\") pod \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " Oct 31 00:20:09.806581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806422 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config\") pod \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " Oct 31 00:20:09.806581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806541 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s\" (UniqueName: \"kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s\") pod \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " Oct 31 00:20:09.806581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806556 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp" pod "0dae54b3-dceb-4795-85ec-42f9fb9ff256" (UID: "0dae54b3-dceb-4795-85ec-42f9fb9ff256"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:09.806729 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806583 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert" pod "0dae54b3-dceb-4795-85ec-42f9fb9ff256" (UID: "0dae54b3-dceb-4795-85ec-42f9fb9ff256"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:09.806729 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806601 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config\") pod \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\" (UID: \"0dae54b3-dceb-4795-85ec-42f9fb9ff256\") " Oct 31 00:20:09.806890 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806866 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle" pod "0dae54b3-dceb-4795-85ec-42f9fb9ff256" (UID: "0dae54b3-dceb-4795-85ec-42f9fb9ff256"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:09.806953 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806903 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca" pod "0dae54b3-dceb-4795-85ec-42f9fb9ff256" (UID: "0dae54b3-dceb-4795-85ec-42f9fb9ff256"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:09.807005 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806973 5978 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-oauth-serving-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:09.807005 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.806995 5978 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dae54b3-dceb-4795-85ec-42f9fb9ff256-tmp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:09.807073 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.807025 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config" pod "0dae54b3-dceb-4795-85ec-42f9fb9ff256" (UID: "0dae54b3-dceb-4795-85ec-42f9fb9ff256"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:09.808598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.808571 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert" pod "0dae54b3-dceb-4795-85ec-42f9fb9ff256" (UID: "0dae54b3-dceb-4795-85ec-42f9fb9ff256"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:09.808808 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.808792 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config" pod "0dae54b3-dceb-4795-85ec-42f9fb9ff256" (UID: "0dae54b3-dceb-4795-85ec-42f9fb9ff256"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:09.808860 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.808793 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s" pod "0dae54b3-dceb-4795-85ec-42f9fb9ff256" (UID: "0dae54b3-dceb-4795-85ec-42f9fb9ff256"). InnerVolumeSpecName "kube-api-access-tzj5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:09.907790 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.907746 5978 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-serving-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:09.907790 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.907782 5978 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-trusted-ca-bundle\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:09.907790 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.907797 5978 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:09.908044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.907811 5978 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:09.908044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.907822 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzj5s\" (UniqueName: \"kubernetes.io/projected/0dae54b3-dceb-4795-85ec-42f9fb9ff256-kube-api-access-tzj5s\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:09.908044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:09.907837 5978 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dae54b3-dceb-4795-85ec-42f9fb9ff256-console-oauth-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:10.421928 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:10.421894 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa8cd46-8095-4e93-b23a-0a422a47eab3" path="/var/lib/kubelet/pods/0fa8cd46-8095-4e93-b23a-0a422a47eab3/volumes" Oct 31 00:20:10.645835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:10.645804 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b4c98b6b-w7lch" Oct 31 00:20:10.646326 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:10.645803 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b4c98b6b-w7lch" event={"ID":"0dae54b3-dceb-4795-85ec-42f9fb9ff256","Type":"ContainerDied","Data":"c4ab2cfa8ceb7546c2058faf373f9b324a7aafcaf5d65b102adbdd9bd438e86f"} Oct 31 00:20:10.646326 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:10.645970 5978 scope.go:122] "RemoveContainer" containerID="7dba283abe3a374371a9fe2b35905e6b1cca9b421835f0bddb66506b02d20dbc" Oct 31 00:20:10.696380 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:10.696305 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69b4c98b6b-w7lch"] Oct 31 00:20:10.708431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:10.708399 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69b4c98b6b-w7lch"] Oct 31 00:20:12.422327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:12.422293 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dae54b3-dceb-4795-85ec-42f9fb9ff256" path="/var/lib/kubelet/pods/0dae54b3-dceb-4795-85ec-42f9fb9ff256/volumes" Oct 31 00:20:12.987247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:12.987213 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mswcp" Oct 31 00:20:13.995519 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:13.995476 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:14.011973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:14.011945 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:14.681441 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:14.681412 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:20:14.768626 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:14.768593 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:14.768804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:14.768654 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:14.774987 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:14.774959 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:15.673306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:15.673281 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:20:15.749736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:15.749704 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb49695f-qnzhd"] Oct 31 00:20:17.834904 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:17.834839 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:17.997737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:17.997704 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:19.365957 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:19.365902 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:20.048868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:20.048831 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:20.215126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:20.215086 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:29.548349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:29.548301 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:29.703184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:29.703138 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:32.380785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.380752 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c5bccddfc-p44kw"] Oct 31 00:20:32.381233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.381143 5978 cpu_manager.go:414] "containerMap: removing container" podUID="0dae54b3-dceb-4795-85ec-42f9fb9ff256" containerName="console" Oct 31 00:20:32.381233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.381160 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dae54b3-dceb-4795-85ec-42f9fb9ff256" containerName="console" Oct 31 00:20:32.381233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.381174 5978 cpu_manager.go:414] "containerMap: removing container" podUID="0fa8cd46-8095-4e93-b23a-0a422a47eab3" containerName="console" Oct 31 00:20:32.381233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.381181 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa8cd46-8095-4e93-b23a-0a422a47eab3" containerName="console" Oct 31 00:20:32.381368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.381256 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dae54b3-dceb-4795-85ec-42f9fb9ff256" containerName="console" Oct 31 00:20:32.381368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.381267 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa8cd46-8095-4e93-b23a-0a422a47eab3" containerName="console" Oct 31 00:20:32.384519 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.384498 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.411777 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.411748 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5bccddfc-p44kw"] Oct 31 00:20:32.449672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.449637 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-console-serving-cert\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.449672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.449681 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-console-config\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.449898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.449698 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-trusted-ca-bundle\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.449898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.449805 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-service-ca\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.449898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.449835 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-console-oauth-config\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.449898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.449856 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmc8\" (UniqueName: \"kubernetes.io/projected/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-kube-api-access-6zmc8\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.450022 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.449953 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-oauth-serving-cert\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.450022 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.449972 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-tmp\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.551117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.551084 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-console-oauth-config\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.551117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.551120 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmc8\" (UniqueName: \"kubernetes.io/projected/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-kube-api-access-6zmc8\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.551383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.551151 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-oauth-serving-cert\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.551383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.551168 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-tmp\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.551383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.551216 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-console-serving-cert\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.551383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.551243 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-console-config\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.551383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.551271 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-trusted-ca-bundle\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.551383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.551351 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-service-ca\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.552006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.551979 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-oauth-serving-cert\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.552148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.552118 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-console-config\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.552304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.552279 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-tmp\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.552769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.552749 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-service-ca\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.552898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.552860 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-trusted-ca-bundle\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.553749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.553707 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:32.554323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.554300 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-console-oauth-config\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.554672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.554652 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-console-serving-cert\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.574887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.574850 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmc8\" (UniqueName: \"kubernetes.io/projected/1bc12e8e-e1be-49ee-8a5c-32026f7e656e-kube-api-access-6zmc8\") pod \"console-5c5bccddfc-p44kw\" (UID: \"1bc12e8e-e1be-49ee-8a5c-32026f7e656e\") " pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.694870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.694785 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:32.733061 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.733023 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:32.931210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:32.931167 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:33.002725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:33.002696 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5bccddfc-p44kw"] Oct 31 00:20:33.004149 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:20:33.004123 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc12e8e_e1be_49ee_8a5c_32026f7e656e.slice/crio-b9da733e8440ccd23ce577eec7354ac2597d2d8dcd197f35df0aa401ae03a706 WatchSource:0}: Error finding container b9da733e8440ccd23ce577eec7354ac2597d2d8dcd197f35df0aa401ae03a706: Status 404 returned error can't find the container with id b9da733e8440ccd23ce577eec7354ac2597d2d8dcd197f35df0aa401ae03a706 Oct 31 00:20:33.114990 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:33.114951 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:33.196232 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:20:33.196192 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:20:33.752051 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:33.752015 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5bccddfc-p44kw" event={"ID":"1bc12e8e-e1be-49ee-8a5c-32026f7e656e","Type":"ContainerStarted","Data":"cdb8f943701a138cc97c92d42995fe9bdf1064ef940d37ac7b383be62957ed51"} Oct 31 00:20:33.752484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:33.752064 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5bccddfc-p44kw" event={"ID":"1bc12e8e-e1be-49ee-8a5c-32026f7e656e","Type":"ContainerStarted","Data":"b9da733e8440ccd23ce577eec7354ac2597d2d8dcd197f35df0aa401ae03a706"} Oct 31 00:20:33.809553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:33.809491 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c5bccddfc-p44kw" podStartSLOduration=1.809475717 podStartE2EDuration="1.809475717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:33.797374875 +0000 UTC m=+51.495488655" watchObservedRunningTime="2026-10-31 00:20:33.809475717 +0000 UTC m=+51.507589476" Oct 31 00:20:40.775481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:40.775406 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-fb49695f-qnzhd" podUID="890a1f30-b435-43fd-9f63-a1123bd6e661" containerName="console" containerID="cri-o://d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6" gracePeriod=15 Oct 31 00:20:41.047235 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.047210 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:20:41.131773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.131737 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle\") pod \"890a1f30-b435-43fd-9f63-a1123bd6e661\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " Oct 31 00:20:41.131976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.131785 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config\") pod \"890a1f30-b435-43fd-9f63-a1123bd6e661\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " Oct 31 00:20:41.131976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.131813 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf\" (UniqueName: \"kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf\") pod \"890a1f30-b435-43fd-9f63-a1123bd6e661\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " Oct 31 00:20:41.132096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132013 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert\") pod \"890a1f30-b435-43fd-9f63-a1123bd6e661\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " Oct 31 00:20:41.132148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132131 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert\") pod \"890a1f30-b435-43fd-9f63-a1123bd6e661\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " Oct 31 00:20:41.132208 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132190 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp\" (UniqueName: \"kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp\") pod \"890a1f30-b435-43fd-9f63-a1123bd6e661\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " Oct 31 00:20:41.132271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132195 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle" pod "890a1f30-b435-43fd-9f63-a1123bd6e661" (UID: "890a1f30-b435-43fd-9f63-a1123bd6e661"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:41.132271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132211 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config" pod "890a1f30-b435-43fd-9f63-a1123bd6e661" (UID: "890a1f30-b435-43fd-9f63-a1123bd6e661"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:41.132271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132227 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca\") pod \"890a1f30-b435-43fd-9f63-a1123bd6e661\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " Oct 31 00:20:41.132271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132269 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config\") pod \"890a1f30-b435-43fd-9f63-a1123bd6e661\" (UID: \"890a1f30-b435-43fd-9f63-a1123bd6e661\") " Oct 31 00:20:41.132551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132523 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert" pod "890a1f30-b435-43fd-9f63-a1123bd6e661" (UID: "890a1f30-b435-43fd-9f63-a1123bd6e661"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:41.132551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132541 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp" pod "890a1f30-b435-43fd-9f63-a1123bd6e661" (UID: "890a1f30-b435-43fd-9f63-a1123bd6e661"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:41.132664 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132639 5978 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-trusted-ca-bundle\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:41.132720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132660 5978 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-console-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:41.132720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132678 5978 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-oauth-serving-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:41.132720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132692 5978 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890a1f30-b435-43fd-9f63-a1123bd6e661-tmp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:41.133019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.132996 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca" pod "890a1f30-b435-43fd-9f63-a1123bd6e661" (UID: "890a1f30-b435-43fd-9f63-a1123bd6e661"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:41.134218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.134187 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert" pod "890a1f30-b435-43fd-9f63-a1123bd6e661" (UID: "890a1f30-b435-43fd-9f63-a1123bd6e661"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:41.134340 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.134242 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf" pod "890a1f30-b435-43fd-9f63-a1123bd6e661" (UID: "890a1f30-b435-43fd-9f63-a1123bd6e661"). InnerVolumeSpecName "kube-api-access-9hhtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:41.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.134865 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config" pod "890a1f30-b435-43fd-9f63-a1123bd6e661" (UID: "890a1f30-b435-43fd-9f63-a1123bd6e661"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:41.233273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.233230 5978 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/890a1f30-b435-43fd-9f63-a1123bd6e661-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:41.233273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.233262 5978 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-oauth-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:41.233273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.233279 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hhtf\" (UniqueName: \"kubernetes.io/projected/890a1f30-b435-43fd-9f63-a1123bd6e661-kube-api-access-9hhtf\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:41.233557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.233292 5978 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/890a1f30-b435-43fd-9f63-a1123bd6e661-console-serving-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:20:41.363827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.363791 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:41.788186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.788099 5978 generic.go:356] "Generic (PLEG): container finished" podID="890a1f30-b435-43fd-9f63-a1123bd6e661" containerID="d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6" exitCode=0 Oct 31 00:20:41.788186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.788152 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb49695f-qnzhd" Oct 31 00:20:41.788708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.788152 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb49695f-qnzhd" event={"ID":"890a1f30-b435-43fd-9f63-a1123bd6e661","Type":"ContainerDied","Data":"d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6"} Oct 31 00:20:41.788708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.788273 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb49695f-qnzhd" event={"ID":"890a1f30-b435-43fd-9f63-a1123bd6e661","Type":"ContainerDied","Data":"738d74e83e8dfae45a6748aabc87ca0bbcc8e5a3fd622d27c4a27e6907ede597"} Oct 31 00:20:41.788708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.788301 5978 scope.go:122] "RemoveContainer" containerID="d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6" Oct 31 00:20:41.797424 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.797404 5978 scope.go:122] "RemoveContainer" containerID="d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6" Oct 31 00:20:41.797734 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:20:41.797715 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6\": container with ID starting with d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6 not found: ID does not exist" containerID="d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6" Oct 31 00:20:41.797796 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.797744 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6"} err="failed to get container status \"d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6\": rpc error: code = NotFound desc = could not find container \"d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6\": container with ID starting with d9446699e8a2b295b898a04cbe2785da4ba19135ef7ed2ab1955bf5bd5587be6 not found: ID does not exist" Oct 31 00:20:41.839311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.839280 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb49695f-qnzhd"] Oct 31 00:20:41.851850 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:41.851819 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fb49695f-qnzhd"] Oct 31 00:20:42.422661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:42.422628 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890a1f30-b435-43fd-9f63-a1123bd6e661" path="/var/lib/kubelet/pods/890a1f30-b435-43fd-9f63-a1123bd6e661/volumes" Oct 31 00:20:42.695127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:42.695041 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:42.695283 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:42.695138 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:42.699760 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:42.699732 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:42.795784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:42.795756 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c5bccddfc-p44kw" Oct 31 00:20:42.890158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:20:42.890124 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68467f5bb4-tgfc8"] Oct 31 00:21:03.179425 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:21:03.179382 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:21:07.909817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:07.909762 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-68467f5bb4-tgfc8" podUID="54afbbc8-2da3-4771-9adf-917639334382" containerName="console" containerID="cri-o://08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d" gracePeriod=15 Oct 31 00:21:08.160220 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.160161 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:21:08.284487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.284440 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-serving-cert\") pod \"54afbbc8-2da3-4771-9adf-917639334382\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " Oct 31 00:21:08.284696 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.284514 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-trusted-ca-bundle\") pod \"54afbbc8-2da3-4771-9adf-917639334382\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " Oct 31 00:21:08.284696 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.284561 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-service-ca\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-service-ca\") pod \"54afbbc8-2da3-4771-9adf-917639334382\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " Oct 31 00:21:08.284696 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.284589 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-oauth-serving-cert\") pod \"54afbbc8-2da3-4771-9adf-917639334382\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " Oct 31 00:21:08.284696 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.284632 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54afbbc8-2da3-4771-9adf-917639334382-kube-api-access-tvgwn\" (UniqueName: \"kubernetes.io/projected/54afbbc8-2da3-4771-9adf-917639334382-kube-api-access-tvgwn\") pod \"54afbbc8-2da3-4771-9adf-917639334382\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " Oct 31 00:21:08.284934 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.284728 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54afbbc8-2da3-4771-9adf-917639334382-tmp\" (UniqueName: \"kubernetes.io/empty-dir/54afbbc8-2da3-4771-9adf-917639334382-tmp\") pod \"54afbbc8-2da3-4771-9adf-917639334382\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " Oct 31 00:21:08.284934 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.284801 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-console-config\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-console-config\") pod \"54afbbc8-2da3-4771-9adf-917639334382\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " Oct 31 00:21:08.284934 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.284880 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-oauth-config\") pod \"54afbbc8-2da3-4771-9adf-917639334382\" (UID: \"54afbbc8-2da3-4771-9adf-917639334382\") " Oct 31 00:21:08.285083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.284996 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-service-ca" pod "54afbbc8-2da3-4771-9adf-917639334382" (UID: "54afbbc8-2da3-4771-9adf-917639334382"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:08.285135 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.285077 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54afbbc8-2da3-4771-9adf-917639334382-tmp" pod "54afbbc8-2da3-4771-9adf-917639334382" (UID: "54afbbc8-2da3-4771-9adf-917639334382"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:21:08.285135 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.285114 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-trusted-ca-bundle" pod "54afbbc8-2da3-4771-9adf-917639334382" (UID: "54afbbc8-2da3-4771-9adf-917639334382"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:08.285232 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.285127 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-oauth-serving-cert" pod "54afbbc8-2da3-4771-9adf-917639334382" (UID: "54afbbc8-2da3-4771-9adf-917639334382"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:08.285343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.285324 5978 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54afbbc8-2da3-4771-9adf-917639334382-tmp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:21:08.285412 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.285349 5978 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-trusted-ca-bundle\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:21:08.285412 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.285364 5978 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:21:08.285412 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.285378 5978 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-oauth-serving-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:21:08.285412 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.285332 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-console-config" pod "54afbbc8-2da3-4771-9adf-917639334382" (UID: "54afbbc8-2da3-4771-9adf-917639334382"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:08.286952 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.286933 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-oauth-config" pod "54afbbc8-2da3-4771-9adf-917639334382" (UID: "54afbbc8-2da3-4771-9adf-917639334382"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:08.287191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.287162 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-serving-cert" pod "54afbbc8-2da3-4771-9adf-917639334382" (UID: "54afbbc8-2da3-4771-9adf-917639334382"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:08.287284 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.287170 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54afbbc8-2da3-4771-9adf-917639334382-kube-api-access-tvgwn" pod "54afbbc8-2da3-4771-9adf-917639334382" (UID: "54afbbc8-2da3-4771-9adf-917639334382"). InnerVolumeSpecName "kube-api-access-tvgwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:08.386104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.386068 5978 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54afbbc8-2da3-4771-9adf-917639334382-console-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:21:08.386104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.386095 5978 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-oauth-config\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:21:08.386104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.386107 5978 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54afbbc8-2da3-4771-9adf-917639334382-console-serving-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:21:08.386334 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.386119 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvgwn\" (UniqueName: \"kubernetes.io/projected/54afbbc8-2da3-4771-9adf-917639334382-kube-api-access-tvgwn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:21:08.884609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.884568 5978 generic.go:356] "Generic (PLEG): container finished" podID="54afbbc8-2da3-4771-9adf-917639334382" containerID="08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d" exitCode=0 Oct 31 00:21:08.884774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.884620 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68467f5bb4-tgfc8" event={"ID":"54afbbc8-2da3-4771-9adf-917639334382","Type":"ContainerDied","Data":"08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d"} Oct 31 00:21:08.884774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.884639 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68467f5bb4-tgfc8" Oct 31 00:21:08.884774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.884699 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68467f5bb4-tgfc8" event={"ID":"54afbbc8-2da3-4771-9adf-917639334382","Type":"ContainerDied","Data":"60e0e2378ac0eedcf46f297d4e16fffbefcf0ff25b9d73acb09b2a4e7f430b10"} Oct 31 00:21:08.884774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.884726 5978 scope.go:122] "RemoveContainer" containerID="08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d" Oct 31 00:21:08.892516 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.892499 5978 scope.go:122] "RemoveContainer" containerID="08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d" Oct 31 00:21:08.892768 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:21:08.892750 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d\": container with ID starting with 08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d not found: ID does not exist" containerID="08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d" Oct 31 00:21:08.892835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.892777 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d"} err="failed to get container status \"08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d\": rpc error: code = NotFound desc = could not find container \"08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d\": container with ID starting with 08758ebd74ab9990adf373e65243e4d9ab104b1d8ef10632d445178ed00bef4d not found: ID does not exist" Oct 31 00:21:08.935510 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.935478 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68467f5bb4-tgfc8"] Oct 31 00:21:08.947776 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:08.947747 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68467f5bb4-tgfc8"] Oct 31 00:21:10.422563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:10.422528 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54afbbc8-2da3-4771-9adf-917639334382" path="/var/lib/kubelet/pods/54afbbc8-2da3-4771-9adf-917639334382/volumes" Oct 31 00:21:31.907172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:31.907132 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:32.072835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:32.072800 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:33.180088 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:21:33.180052 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:21:41.492955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:41.492919 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:42.638741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:42.638698 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:42.791080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:21:42.790987 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:02.031721 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:02.031687 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:03.179765 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:22:03.179727 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:22:03.783143 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:03.783093 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:04.023141 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:04.023105 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.537189 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:05.537153 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.685294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:05.685257 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:06.956253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:06.956220 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:09.831158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:09.831126 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:09.977846 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:09.977806 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:11.087242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.087207 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:11.265034 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.264995 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:11.385138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.385057 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5"] Oct 31 00:22:11.385443 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.385430 5978 cpu_manager.go:414] "containerMap: removing container" podUID="890a1f30-b435-43fd-9f63-a1123bd6e661" containerName="console" Oct 31 00:22:11.385513 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.385445 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890a1f30-b435-43fd-9f63-a1123bd6e661" containerName="console" Oct 31 00:22:11.385513 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.385486 5978 cpu_manager.go:414] "containerMap: removing container" podUID="54afbbc8-2da3-4771-9adf-917639334382" containerName="console" Oct 31 00:22:11.385513 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.385493 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54afbbc8-2da3-4771-9adf-917639334382" containerName="console" Oct 31 00:22:11.385613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.385545 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="890a1f30-b435-43fd-9f63-a1123bd6e661" containerName="console" Oct 31 00:22:11.385613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.385555 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="54afbbc8-2da3-4771-9adf-917639334382" containerName="console" Oct 31 00:22:11.388626 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.388609 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.418773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.418736 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:22:11.418912 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.418837 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:22:11.418912 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.418855 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-l9j5t\"" Oct 31 00:22:11.437714 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.437682 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5"] Oct 31 00:22:11.441176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.441144 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m5ql\" (UniqueName: \"kubernetes.io/projected/56d91213-dabe-4cf0-a61a-fabdd81fe326-kube-api-access-7m5ql\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.441320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.441180 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.441320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.441205 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.448411 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.448383 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:11.542701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.542661 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m5ql\" (UniqueName: \"kubernetes.io/projected/56d91213-dabe-4cf0-a61a-fabdd81fe326-kube-api-access-7m5ql\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.542701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.542705 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.542952 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.542735 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.543149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.543125 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.543218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.543142 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.566064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.566036 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m5ql\" (UniqueName: \"kubernetes.io/projected/56d91213-dabe-4cf0-a61a-fabdd81fe326-kube-api-access-7m5ql\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.704932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.704854 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:11.775089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.773582 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn"] Oct 31 00:22:11.777906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.777508 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:11.795210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.795190 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-85jgg\"" Oct 31 00:22:11.805724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.805699 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn"] Oct 31 00:22:11.845113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.845071 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcq4n\" (UniqueName: \"kubernetes.io/projected/838332cb-b471-40a8-b8c1-b93d898046f9-kube-api-access-mcq4n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:11.845113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.845114 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:11.845333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.845136 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:11.945728 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.945692 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcq4n\" (UniqueName: \"kubernetes.io/projected/838332cb-b471-40a8-b8c1-b93d898046f9-kube-api-access-mcq4n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:11.945879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.945732 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:11.945879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.945758 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:11.946122 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.946105 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:11.946168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.946125 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:11.970145 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:11.970070 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcq4n\" (UniqueName: \"kubernetes.io/projected/838332cb-b471-40a8-b8c1-b93d898046f9-kube-api-access-mcq4n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:12.017933 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:12.017910 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5"] Oct 31 00:22:12.019379 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:22:12.019352 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d91213_dabe_4cf0_a61a_fabdd81fe326.slice/crio-e6cd3e1ccc366e5f44018ed342e162eac12c3abb1be85e3bf78db5cc8151c1b0 WatchSource:0}: Error finding container e6cd3e1ccc366e5f44018ed342e162eac12c3abb1be85e3bf78db5cc8151c1b0: Status 404 returned error can't find the container with id e6cd3e1ccc366e5f44018ed342e162eac12c3abb1be85e3bf78db5cc8151c1b0 Oct 31 00:22:12.021380 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:12.021361 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:22:12.093981 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:12.093949 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:12.115728 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:12.115697 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" event={"ID":"56d91213-dabe-4cf0-a61a-fabdd81fe326","Type":"ContainerStarted","Data":"e6cd3e1ccc366e5f44018ed342e162eac12c3abb1be85e3bf78db5cc8151c1b0"} Oct 31 00:22:12.391778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:12.391745 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn"] Oct 31 00:22:12.392116 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:22:12.392095 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838332cb_b471_40a8_b8c1_b93d898046f9.slice/crio-794a7cefd51dc2cd5ebd44242a16e9ded86f300671b041e6770a1cc8fa7c6d88 WatchSource:0}: Error finding container 794a7cefd51dc2cd5ebd44242a16e9ded86f300671b041e6770a1cc8fa7c6d88: Status 404 returned error can't find the container with id 794a7cefd51dc2cd5ebd44242a16e9ded86f300671b041e6770a1cc8fa7c6d88 Oct 31 00:22:13.122002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:13.121932 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" event={"ID":"838332cb-b471-40a8-b8c1-b93d898046f9","Type":"ContainerStarted","Data":"794a7cefd51dc2cd5ebd44242a16e9ded86f300671b041e6770a1cc8fa7c6d88"} Oct 31 00:22:14.745136 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:14.745080 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:18.147823 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:18.147791 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" event={"ID":"56d91213-dabe-4cf0-a61a-fabdd81fe326","Type":"ContainerStarted","Data":"757622432fc6eca54e2d8e362f4f3793adb60dab5d6e7cb4326811a1720c75b6"} Oct 31 00:22:19.153044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:19.153005 5978 generic.go:356] "Generic (PLEG): container finished" podID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerID="757622432fc6eca54e2d8e362f4f3793adb60dab5d6e7cb4326811a1720c75b6" exitCode=0 Oct 31 00:22:19.153515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:19.153099 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" event={"ID":"56d91213-dabe-4cf0-a61a-fabdd81fe326","Type":"ContainerDied","Data":"757622432fc6eca54e2d8e362f4f3793adb60dab5d6e7cb4326811a1720c75b6"} Oct 31 00:22:19.154830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:19.154799 5978 generic.go:356] "Generic (PLEG): container finished" podID="838332cb-b471-40a8-b8c1-b93d898046f9" containerID="7e5ba6ada1656746e729de22884e04f66acc880d356702b93dc90354fda378d4" exitCode=0 Oct 31 00:22:19.154932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:19.154832 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" event={"ID":"838332cb-b471-40a8-b8c1-b93d898046f9","Type":"ContainerDied","Data":"7e5ba6ada1656746e729de22884e04f66acc880d356702b93dc90354fda378d4"} Oct 31 00:22:20.161129 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:20.161093 5978 generic.go:356] "Generic (PLEG): container finished" podID="838332cb-b471-40a8-b8c1-b93d898046f9" containerID="deb1a4102f8dfa960ffdd7d32271d0fca08b1d8f8bc1c1b79696bec1d19d81bf" exitCode=0 Oct 31 00:22:20.161566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:20.161183 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" event={"ID":"838332cb-b471-40a8-b8c1-b93d898046f9","Type":"ContainerDied","Data":"deb1a4102f8dfa960ffdd7d32271d0fca08b1d8f8bc1c1b79696bec1d19d81bf"} Oct 31 00:22:22.171777 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:22.171742 5978 generic.go:356] "Generic (PLEG): container finished" podID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerID="3c4f817fe55ea691659579991c900903a381fc7c8685bc65020b117a9932fa37" exitCode=0 Oct 31 00:22:22.172221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:22.171805 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" event={"ID":"56d91213-dabe-4cf0-a61a-fabdd81fe326","Type":"ContainerDied","Data":"3c4f817fe55ea691659579991c900903a381fc7c8685bc65020b117a9932fa37"} Oct 31 00:22:27.198210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:27.198166 5978 generic.go:356] "Generic (PLEG): container finished" podID="838332cb-b471-40a8-b8c1-b93d898046f9" containerID="e4da6a8aa53479ce906e2dc0236999615efc8b461c02bf217d041dcc4c7bf145" exitCode=0 Oct 31 00:22:27.198743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:27.198260 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" event={"ID":"838332cb-b471-40a8-b8c1-b93d898046f9","Type":"ContainerDied","Data":"e4da6a8aa53479ce906e2dc0236999615efc8b461c02bf217d041dcc4c7bf145"} Oct 31 00:22:28.354316 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.354292 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:28.514301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.514216 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-util\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-util\") pod \"838332cb-b471-40a8-b8c1-b93d898046f9\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " Oct 31 00:22:28.514511 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.514304 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838332cb-b471-40a8-b8c1-b93d898046f9-kube-api-access-mcq4n\" (UniqueName: \"kubernetes.io/projected/838332cb-b471-40a8-b8c1-b93d898046f9-kube-api-access-mcq4n\") pod \"838332cb-b471-40a8-b8c1-b93d898046f9\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " Oct 31 00:22:28.514511 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.514350 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-bundle\") pod \"838332cb-b471-40a8-b8c1-b93d898046f9\" (UID: \"838332cb-b471-40a8-b8c1-b93d898046f9\") " Oct 31 00:22:28.514929 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.514901 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-bundle" pod "838332cb-b471-40a8-b8c1-b93d898046f9" (UID: "838332cb-b471-40a8-b8c1-b93d898046f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:28.516581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.516546 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838332cb-b471-40a8-b8c1-b93d898046f9-kube-api-access-mcq4n" pod "838332cb-b471-40a8-b8c1-b93d898046f9" (UID: "838332cb-b471-40a8-b8c1-b93d898046f9"). InnerVolumeSpecName "kube-api-access-mcq4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:28.519587 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.519561 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-util" pod "838332cb-b471-40a8-b8c1-b93d898046f9" (UID: "838332cb-b471-40a8-b8c1-b93d898046f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:28.615083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.615045 5978 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-bundle\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:22:28.615083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.615077 5978 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/838332cb-b471-40a8-b8c1-b93d898046f9-util\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:22:28.615083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:28.615087 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcq4n\" (UniqueName: \"kubernetes.io/projected/838332cb-b471-40a8-b8c1-b93d898046f9-kube-api-access-mcq4n\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:22:29.208611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:29.208577 5978 generic.go:356] "Generic (PLEG): container finished" podID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerID="ce378ecbf55b8a7428f3eabd8627e89448a9e86077783ef282daf0ef577772a7" exitCode=0 Oct 31 00:22:29.208806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:29.208653 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" event={"ID":"56d91213-dabe-4cf0-a61a-fabdd81fe326","Type":"ContainerDied","Data":"ce378ecbf55b8a7428f3eabd8627e89448a9e86077783ef282daf0ef577772a7"} Oct 31 00:22:29.210722 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:29.210695 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" event={"ID":"838332cb-b471-40a8-b8c1-b93d898046f9","Type":"ContainerDied","Data":"794a7cefd51dc2cd5ebd44242a16e9ded86f300671b041e6770a1cc8fa7c6d88"} Oct 31 00:22:29.210858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:29.210726 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcjwn" Oct 31 00:22:29.210858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:29.210739 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="794a7cefd51dc2cd5ebd44242a16e9ded86f300671b041e6770a1cc8fa7c6d88" Oct 31 00:22:30.341925 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.341894 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:30.531516 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.531400 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d91213-dabe-4cf0-a61a-fabdd81fe326-kube-api-access-7m5ql\" (UniqueName: \"kubernetes.io/projected/56d91213-dabe-4cf0-a61a-fabdd81fe326-kube-api-access-7m5ql\") pod \"56d91213-dabe-4cf0-a61a-fabdd81fe326\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " Oct 31 00:22:30.531685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.531530 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-bundle\") pod \"56d91213-dabe-4cf0-a61a-fabdd81fe326\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " Oct 31 00:22:30.531685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.531573 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-util\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-util\") pod \"56d91213-dabe-4cf0-a61a-fabdd81fe326\" (UID: \"56d91213-dabe-4cf0-a61a-fabdd81fe326\") " Oct 31 00:22:30.532193 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.532166 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-bundle" pod "56d91213-dabe-4cf0-a61a-fabdd81fe326" (UID: "56d91213-dabe-4cf0-a61a-fabdd81fe326"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:30.533445 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.533423 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d91213-dabe-4cf0-a61a-fabdd81fe326-kube-api-access-7m5ql" pod "56d91213-dabe-4cf0-a61a-fabdd81fe326" (UID: "56d91213-dabe-4cf0-a61a-fabdd81fe326"). InnerVolumeSpecName "kube-api-access-7m5ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:30.535805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.535777 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-util" pod "56d91213-dabe-4cf0-a61a-fabdd81fe326" (UID: "56d91213-dabe-4cf0-a61a-fabdd81fe326"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:30.633034 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.632991 5978 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-bundle\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:22:30.633034 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.633026 5978 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56d91213-dabe-4cf0-a61a-fabdd81fe326-util\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:22:30.633034 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:30.633039 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m5ql\" (UniqueName: \"kubernetes.io/projected/56d91213-dabe-4cf0-a61a-fabdd81fe326-kube-api-access-7m5ql\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:22:31.221259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:31.221227 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" Oct 31 00:22:31.221472 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:31.221225 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cplgr5" event={"ID":"56d91213-dabe-4cf0-a61a-fabdd81fe326","Type":"ContainerDied","Data":"e6cd3e1ccc366e5f44018ed342e162eac12c3abb1be85e3bf78db5cc8151c1b0"} Oct 31 00:22:31.221472 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:31.221351 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6cd3e1ccc366e5f44018ed342e162eac12c3abb1be85e3bf78db5cc8151c1b0" Oct 31 00:22:32.740398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:32.740359 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:33.188183 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:22:33.188142 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:22:34.539949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:34.539907 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:35.510318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510287 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2"] Oct 31 00:22:35.510712 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510696 5978 cpu_manager.go:414] "containerMap: removing container" podUID="838332cb-b471-40a8-b8c1-b93d898046f9" containerName="pull" Oct 31 00:22:35.510767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510713 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838332cb-b471-40a8-b8c1-b93d898046f9" containerName="pull" Oct 31 00:22:35.510767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510725 5978 cpu_manager.go:414] "containerMap: removing container" podUID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerName="pull" Oct 31 00:22:35.510767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510730 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerName="pull" Oct 31 00:22:35.510767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510740 5978 cpu_manager.go:414] "containerMap: removing container" podUID="838332cb-b471-40a8-b8c1-b93d898046f9" containerName="extract" Oct 31 00:22:35.510767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510745 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838332cb-b471-40a8-b8c1-b93d898046f9" containerName="extract" Oct 31 00:22:35.510930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510775 5978 cpu_manager.go:414] "containerMap: removing container" podUID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerName="extract" Oct 31 00:22:35.510930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510780 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerName="extract" Oct 31 00:22:35.510930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510795 5978 cpu_manager.go:414] "containerMap: removing container" podUID="838332cb-b471-40a8-b8c1-b93d898046f9" containerName="util" Oct 31 00:22:35.510930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510799 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838332cb-b471-40a8-b8c1-b93d898046f9" containerName="util" Oct 31 00:22:35.510930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510809 5978 cpu_manager.go:414] "containerMap: removing container" podUID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerName="util" Oct 31 00:22:35.510930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510814 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerName="util" Oct 31 00:22:35.510930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510872 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d91213-dabe-4cf0-a61a-fabdd81fe326" containerName="extract" Oct 31 00:22:35.510930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.510881 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="838332cb-b471-40a8-b8c1-b93d898046f9" containerName="extract" Oct 31 00:22:35.553149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.553110 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2"] Oct 31 00:22:35.553609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.553215 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" Oct 31 00:22:35.569889 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.569861 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:22:35.570067 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.569991 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 00:22:35.570167 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.570153 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5l75f\"" Oct 31 00:22:35.574540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.574520 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjj6q\" (UniqueName: \"kubernetes.io/projected/a96b52ec-eab3-41b2-b424-e330d96d5a99-kube-api-access-hjj6q\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2\" (UID: \"a96b52ec-eab3-41b2-b424-e330d96d5a99\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" Oct 31 00:22:35.574622 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.574589 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a96b52ec-eab3-41b2-b424-e330d96d5a99-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2\" (UID: \"a96b52ec-eab3-41b2-b424-e330d96d5a99\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" Oct 31 00:22:35.675607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.675564 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjj6q\" (UniqueName: \"kubernetes.io/projected/a96b52ec-eab3-41b2-b424-e330d96d5a99-kube-api-access-hjj6q\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2\" (UID: \"a96b52ec-eab3-41b2-b424-e330d96d5a99\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" Oct 31 00:22:35.675793 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.675663 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a96b52ec-eab3-41b2-b424-e330d96d5a99-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2\" (UID: \"a96b52ec-eab3-41b2-b424-e330d96d5a99\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" Oct 31 00:22:35.676021 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.676005 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a96b52ec-eab3-41b2-b424-e330d96d5a99-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2\" (UID: \"a96b52ec-eab3-41b2-b424-e330d96d5a99\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" Oct 31 00:22:35.698939 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.698901 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjj6q\" (UniqueName: \"kubernetes.io/projected/a96b52ec-eab3-41b2-b424-e330d96d5a99-kube-api-access-hjj6q\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2\" (UID: \"a96b52ec-eab3-41b2-b424-e330d96d5a99\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" Oct 31 00:22:35.877699 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:35.877656 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" Oct 31 00:22:36.230405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:36.230383 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2"] Oct 31 00:22:36.232943 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:22:36.232913 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96b52ec_eab3_41b2_b424_e330d96d5a99.slice/crio-ee99c6fb4b7470cdd261bec5513615293db8fc62dc9c2889465cb28c3db78388 WatchSource:0}: Error finding container ee99c6fb4b7470cdd261bec5513615293db8fc62dc9c2889465cb28c3db78388: Status 404 returned error can't find the container with id ee99c6fb4b7470cdd261bec5513615293db8fc62dc9c2889465cb28c3db78388 Oct 31 00:22:36.243049 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:36.243020 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" event={"ID":"a96b52ec-eab3-41b2-b424-e330d96d5a99","Type":"ContainerStarted","Data":"ee99c6fb4b7470cdd261bec5513615293db8fc62dc9c2889465cb28c3db78388"} Oct 31 00:22:37.754557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:37.754522 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:37.912677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:37.912640 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:38.165607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:38.165571 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:38.340050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:38.340005 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:39.260515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.260482 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" event={"ID":"a96b52ec-eab3-41b2-b424-e330d96d5a99","Type":"ContainerStarted","Data":"8289835e5828f97183ea2c2cbc62980caffa23e305eed84021c57f8f903058b7"} Oct 31 00:22:39.333572 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.333506 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qtqh2" podStartSLOduration=2.384506324 podStartE2EDuration="4.333487778s" totalImagesPullingTime="1.948981454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:39.322099522 +0000 UTC m=+177.020213288" watchObservedRunningTime="2026-10-31 00:22:39.333487778 +0000 UTC m=+177.031601543" Oct 31 00:22:39.471053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.471021 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:39.636567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.636535 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b"] Oct 31 00:22:39.658301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.658271 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.684120 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.684084 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b"] Oct 31 00:22:39.715216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.715186 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl48j\" (UniqueName: \"kubernetes.io/projected/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-kube-api-access-tl48j\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.715396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.715230 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-metrics-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.715396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.715246 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-manifests\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.715396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.715311 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.715396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.715356 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-webhook-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.715619 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.715496 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-tmp\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.731522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.731464 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 00:22:39.732145 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.732127 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 00:22:39.732298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.732239 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 00:22:39.733588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.733558 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2v5hv\"" Oct 31 00:22:39.733715 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.733640 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 00:22:39.733818 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.733803 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 00:22:39.816409 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.816374 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl48j\" (UniqueName: \"kubernetes.io/projected/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-kube-api-access-tl48j\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.816628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.816430 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-metrics-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.816628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.816581 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-manifests\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.816628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.816621 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.816872 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.816649 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-webhook-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.816872 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.816684 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-tmp\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.817216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.817184 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-manifests\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.818905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.818884 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-tmp\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.819129 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.819113 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.819200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.819184 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-metrics-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.819258 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.819237 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-webhook-cert\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.864694 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.864657 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl48j\" (UniqueName: \"kubernetes.io/projected/fb5c4a1f-24bb-4f93-90ac-8dc640052e07-kube-api-access-tl48j\") pod \"opendatahub-operator-controller-manager-f465f7bb6-9x87b\" (UID: \"fb5c4a1f-24bb-4f93-90ac-8dc640052e07\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:39.897783 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:39.897674 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:40.004720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.004692 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:40.240854 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.240753 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:40.398860 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.398831 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b"] Oct 31 00:22:40.400430 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:22:40.400403 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5c4a1f_24bb_4f93_90ac_8dc640052e07.slice/crio-a8f4e5569eb1a89cf4e82edf98948da77b3651ef0ad5e829f4df0cb227ccfd7c WatchSource:0}: Error finding container a8f4e5569eb1a89cf4e82edf98948da77b3651ef0ad5e829f4df0cb227ccfd7c: Status 404 returned error can't find the container with id a8f4e5569eb1a89cf4e82edf98948da77b3651ef0ad5e829f4df0cb227ccfd7c Oct 31 00:22:40.488851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.488813 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:40.563380 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.563301 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4"] Oct 31 00:22:40.591706 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.591679 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:40.609520 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:22:40.609480 5978 status_manager.go:1045] "Failed to get status for pod" err="pods \"cert-manager-webhook-5dbbbf75cc-72ff4\" is forbidden: User \"system:node:ip-10-0-129-199.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-129-199.ec2.internal' and this object" podUID="a3a55405-5cba-4424-b346-9b2c8d98b3a0" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:40.625602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.625572 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3a55405-5cba-4424-b346-9b2c8d98b3a0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-72ff4\" (UID: \"a3a55405-5cba-4424-b346-9b2c8d98b3a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:40.625762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.625618 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvpd5\" (UniqueName: \"kubernetes.io/projected/a3a55405-5cba-4424-b346-9b2c8d98b3a0-kube-api-access-fvpd5\") pod \"cert-manager-webhook-5dbbbf75cc-72ff4\" (UID: \"a3a55405-5cba-4424-b346-9b2c8d98b3a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:40.639226 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.639200 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 00:22:40.639374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.639200 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 00:22:40.639374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.639243 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7x4m4\"" Oct 31 00:22:40.662702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.662670 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4"] Oct 31 00:22:40.726461 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.726421 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3a55405-5cba-4424-b346-9b2c8d98b3a0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-72ff4\" (UID: \"a3a55405-5cba-4424-b346-9b2c8d98b3a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:40.726661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.726515 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvpd5\" (UniqueName: \"kubernetes.io/projected/a3a55405-5cba-4424-b346-9b2c8d98b3a0-kube-api-access-fvpd5\") pod \"cert-manager-webhook-5dbbbf75cc-72ff4\" (UID: \"a3a55405-5cba-4424-b346-9b2c8d98b3a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:40.748059 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.748028 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:40.753973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.753940 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3a55405-5cba-4424-b346-9b2c8d98b3a0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-72ff4\" (UID: \"a3a55405-5cba-4424-b346-9b2c8d98b3a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:40.768506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.768480 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvpd5\" (UniqueName: \"kubernetes.io/projected/a3a55405-5cba-4424-b346-9b2c8d98b3a0-kube-api-access-fvpd5\") pod \"cert-manager-webhook-5dbbbf75cc-72ff4\" (UID: \"a3a55405-5cba-4424-b346-9b2c8d98b3a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:40.940152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.940115 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:40.966368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:40.966305 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:41.273267 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:41.273170 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" event={"ID":"fb5c4a1f-24bb-4f93-90ac-8dc640052e07","Type":"ContainerStarted","Data":"a8f4e5569eb1a89cf4e82edf98948da77b3651ef0ad5e829f4df0cb227ccfd7c"} Oct 31 00:22:41.288007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:41.287940 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4"] Oct 31 00:22:41.296230 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:22:41.296196 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3a55405_5cba_4424_b346_9b2c8d98b3a0.slice/crio-559eaf0ec22373867d8ff746e7f0ce10234b55b527e50e4f0864170acd506bce WatchSource:0}: Error finding container 559eaf0ec22373867d8ff746e7f0ce10234b55b527e50e4f0864170acd506bce: Status 404 returned error can't find the container with id 559eaf0ec22373867d8ff746e7f0ce10234b55b527e50e4f0864170acd506bce Oct 31 00:22:41.656635 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:41.656602 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:42.279676 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:42.279645 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" event={"ID":"a3a55405-5cba-4424-b346-9b2c8d98b3a0","Type":"ContainerStarted","Data":"559eaf0ec22373867d8ff746e7f0ce10234b55b527e50e4f0864170acd506bce"} Oct 31 00:22:42.643206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:42.643151 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:42.803090 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:42.803049 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:43.177094 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:43.177056 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:43.337104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:43.337053 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:44.291997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:44.291910 5978 generic.go:356] "Generic (PLEG): container finished" podID="fb5c4a1f-24bb-4f93-90ac-8dc640052e07" containerID="4eb49ddc441b890bfadb9629e8daf0a3f2559f6e75c171bf204d1307c9fc4a58" exitCode=0 Oct 31 00:22:44.292443 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:44.292006 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" event={"ID":"fb5c4a1f-24bb-4f93-90ac-8dc640052e07","Type":"ContainerDied","Data":"4eb49ddc441b890bfadb9629e8daf0a3f2559f6e75c171bf204d1307c9fc4a58"} Oct 31 00:22:45.297974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:45.297931 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" event={"ID":"fb5c4a1f-24bb-4f93-90ac-8dc640052e07","Type":"ContainerStarted","Data":"77f07f8acd9ecaa48d37794c35fabbbd07fd51dbabff234c739d8c5ee6b68180"} Oct 31 00:22:45.298546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:45.297998 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:45.299472 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:45.299435 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" event={"ID":"a3a55405-5cba-4424-b346-9b2c8d98b3a0","Type":"ContainerStarted","Data":"d1f995c7901cb0d6dda327c83c2464d789f2bc0bbba9e85ae93a83f5cdc5118b"} Oct 31 00:22:45.299592 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:45.299581 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:45.394870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:45.390568 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" podStartSLOduration=2.66374189 podStartE2EDuration="6.390549551s" totalImagesPullingTime="3.199727933s" totalInitContainerRuntime="527.079728ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:45.388776503 +0000 UTC m=+183.086890307" watchObservedRunningTime="2026-10-31 00:22:45.390549551 +0000 UTC m=+183.088663317" Oct 31 00:22:45.438237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:45.438183 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" podStartSLOduration=1.6393433050000001 podStartE2EDuration="5.438167679s" totalImagesPullingTime="3.798824374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:45.436924685 +0000 UTC m=+183.135038461" watchObservedRunningTime="2026-10-31 00:22:45.438167679 +0000 UTC m=+183.136281443" Oct 31 00:22:45.725812 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:45.725715 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:45.879623 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:45.879585 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:47.339144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:47.339106 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:47.826020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:47.825990 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm"] Oct 31 00:22:47.828415 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:47.828392 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" Oct 31 00:22:47.846639 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:47.846609 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-z4t9v\"" Oct 31 00:22:47.859635 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:47.859605 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm"] Oct 31 00:22:47.898684 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:47.898641 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rbq\" (UniqueName: \"kubernetes.io/projected/167bced1-8e4b-4ea8-9ea8-fe42e6975700-kube-api-access-p8rbq\") pod \"cert-manager-cainjector-5c57bbd4fc-mg9zm\" (UID: \"167bced1-8e4b-4ea8-9ea8-fe42e6975700\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" Oct 31 00:22:47.898857 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:47.898785 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/167bced1-8e4b-4ea8-9ea8-fe42e6975700-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mg9zm\" (UID: \"167bced1-8e4b-4ea8-9ea8-fe42e6975700\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" Oct 31 00:22:48.000079 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:48.000038 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rbq\" (UniqueName: \"kubernetes.io/projected/167bced1-8e4b-4ea8-9ea8-fe42e6975700-kube-api-access-p8rbq\") pod \"cert-manager-cainjector-5c57bbd4fc-mg9zm\" (UID: \"167bced1-8e4b-4ea8-9ea8-fe42e6975700\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" Oct 31 00:22:48.000258 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:48.000141 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/167bced1-8e4b-4ea8-9ea8-fe42e6975700-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mg9zm\" (UID: \"167bced1-8e4b-4ea8-9ea8-fe42e6975700\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" Oct 31 00:22:48.023335 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:48.023302 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/167bced1-8e4b-4ea8-9ea8-fe42e6975700-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mg9zm\" (UID: \"167bced1-8e4b-4ea8-9ea8-fe42e6975700\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" Oct 31 00:22:48.023606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:48.023583 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rbq\" (UniqueName: \"kubernetes.io/projected/167bced1-8e4b-4ea8-9ea8-fe42e6975700-kube-api-access-p8rbq\") pod \"cert-manager-cainjector-5c57bbd4fc-mg9zm\" (UID: \"167bced1-8e4b-4ea8-9ea8-fe42e6975700\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" Oct 31 00:22:48.138399 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:48.138309 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" Oct 31 00:22:48.353030 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:48.352974 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:48.453749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:48.453718 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm"] Oct 31 00:22:48.455006 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:22:48.454976 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod167bced1_8e4b_4ea8_9ea8_fe42e6975700.slice/crio-6534f44795493afa6783a619868df3ef6caee7743d9cca20bf22cf806867c7b3 WatchSource:0}: Error finding container 6534f44795493afa6783a619868df3ef6caee7743d9cca20bf22cf806867c7b3: Status 404 returned error can't find the container with id 6534f44795493afa6783a619868df3ef6caee7743d9cca20bf22cf806867c7b3 Oct 31 00:22:49.318960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:49.318927 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" event={"ID":"167bced1-8e4b-4ea8-9ea8-fe42e6975700","Type":"ContainerStarted","Data":"586a484f704ffeeb56f60f7fa33c3d25117531ce9bde6b653f00162f07c5afb2"} Oct 31 00:22:49.318960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:49.318965 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" event={"ID":"167bced1-8e4b-4ea8-9ea8-fe42e6975700","Type":"ContainerStarted","Data":"6534f44795493afa6783a619868df3ef6caee7743d9cca20bf22cf806867c7b3"} Oct 31 00:22:49.362951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:49.362884 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mg9zm" podStartSLOduration=2.362869841 podStartE2EDuration="2.362869841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:49.362588941 +0000 UTC m=+187.060702708" watchObservedRunningTime="2026-10-31 00:22:49.362869841 +0000 UTC m=+187.060983606" Oct 31 00:22:49.556414 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:49.556375 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:49.719851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:49.719770 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:49.870787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:49.870749 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:51.305967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:51.305936 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" Oct 31 00:22:53.984117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:53.984033 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:54.192606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:54.192572 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:54.430674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:54.430631 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:55.459383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:55.459343 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:56.306079 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:56.306049 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f465f7bb6-9x87b" Oct 31 00:22:57.047815 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.047782 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd"] Oct 31 00:22:57.050049 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.050033 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" Oct 31 00:22:57.065420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.065397 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:22:57.065420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.065404 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 00:22:57.065593 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.065406 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-vtwd8\"" Oct 31 00:22:57.072789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.072766 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd"] Oct 31 00:22:57.189753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.189713 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9819f72a-606e-40aa-b534-f28ccff796f7-tmp\") pod \"jobset-operator-674cd59dc5-mlpkd\" (UID: \"9819f72a-606e-40aa-b534-f28ccff796f7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" Oct 31 00:22:57.189978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.189782 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l7ng\" (UniqueName: \"kubernetes.io/projected/9819f72a-606e-40aa-b534-f28ccff796f7-kube-api-access-8l7ng\") pod \"jobset-operator-674cd59dc5-mlpkd\" (UID: \"9819f72a-606e-40aa-b534-f28ccff796f7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" Oct 31 00:22:57.291015 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.290974 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l7ng\" (UniqueName: \"kubernetes.io/projected/9819f72a-606e-40aa-b534-f28ccff796f7-kube-api-access-8l7ng\") pod \"jobset-operator-674cd59dc5-mlpkd\" (UID: \"9819f72a-606e-40aa-b534-f28ccff796f7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" Oct 31 00:22:57.291207 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.291072 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9819f72a-606e-40aa-b534-f28ccff796f7-tmp\") pod \"jobset-operator-674cd59dc5-mlpkd\" (UID: \"9819f72a-606e-40aa-b534-f28ccff796f7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" Oct 31 00:22:57.291511 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.291491 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9819f72a-606e-40aa-b534-f28ccff796f7-tmp\") pod \"jobset-operator-674cd59dc5-mlpkd\" (UID: \"9819f72a-606e-40aa-b534-f28ccff796f7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" Oct 31 00:22:57.315744 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.315677 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l7ng\" (UniqueName: \"kubernetes.io/projected/9819f72a-606e-40aa-b534-f28ccff796f7-kube-api-access-8l7ng\") pod \"jobset-operator-674cd59dc5-mlpkd\" (UID: \"9819f72a-606e-40aa-b534-f28ccff796f7\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" Oct 31 00:22:57.360638 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.360610 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" Oct 31 00:22:57.669037 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:57.669009 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd"] Oct 31 00:22:57.669862 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:22:57.669837 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9819f72a_606e_40aa_b534_f28ccff796f7.slice/crio-7712787821ff6a811681b2964385de245ab4d6efd96384dc99482048e8c12790 WatchSource:0}: Error finding container 7712787821ff6a811681b2964385de245ab4d6efd96384dc99482048e8c12790: Status 404 returned error can't find the container with id 7712787821ff6a811681b2964385de245ab4d6efd96384dc99482048e8c12790 Oct 31 00:22:58.174737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:58.174706 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:58.364522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:58.364484 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" event={"ID":"9819f72a-606e-40aa-b534-f28ccff796f7","Type":"ContainerStarted","Data":"7712787821ff6a811681b2964385de245ab4d6efd96384dc99482048e8c12790"} Oct 31 00:22:58.535766 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:22:58.535671 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:00.716723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:00.716688 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:01.380117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:01.380083 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" event={"ID":"9819f72a-606e-40aa-b534-f28ccff796f7","Type":"ContainerStarted","Data":"7fe4ecaa2f86fc47d1b0f8f2c8d01ea23074ed92176fc5dd83a5a680a9a08e8e"} Oct 31 00:23:01.429313 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:01.429260 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mlpkd" podStartSLOduration=1.597288716 podStartE2EDuration="4.429242322s" totalImagesPullingTime="2.831953606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:01.428855559 +0000 UTC m=+199.126969323" watchObservedRunningTime="2026-10-31 00:23:01.429242322 +0000 UTC m=+199.127356086" Oct 31 00:23:01.809168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:01.809080 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:02.052770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:02.052736 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:02.365083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:02.365031 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:02.556174 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:02.556143 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:03.057537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:03.057495 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:03.185187 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:23:03.185146 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:23:03.457582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:03.457547 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:09.928317 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:09.928283 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:10.085172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:10.085124 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:10.335942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:10.335911 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:10.505871 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:10.505829 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:13.086617 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.086582 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:13.292845 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.292804 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:13.611876 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.611841 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:13.693231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.693199 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv"] Oct 31 00:23:13.711221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.711187 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.730066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.730036 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv"] Oct 31 00:23:13.740644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.740603 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:23:13.740644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.740626 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:23:13.740891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.740650 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 00:23:13.740891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.740726 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 00:23:13.741386 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.741359 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dvh59\"" Oct 31 00:23:13.807251 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.807219 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:13.851686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.851646 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcb321a-95e1-414c-bafc-c819681071a1-tls-certs\") pod \"kube-auth-proxy-94bcdf98f-xh2jv\" (UID: \"fbcb321a-95e1-414c-bafc-c819681071a1\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.851907 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.851697 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbcb321a-95e1-414c-bafc-c819681071a1-tmp\") pod \"kube-auth-proxy-94bcdf98f-xh2jv\" (UID: \"fbcb321a-95e1-414c-bafc-c819681071a1\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.851907 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.851722 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4hg\" (UniqueName: \"kubernetes.io/projected/fbcb321a-95e1-414c-bafc-c819681071a1-kube-api-access-2g4hg\") pod \"kube-auth-proxy-94bcdf98f-xh2jv\" (UID: \"fbcb321a-95e1-414c-bafc-c819681071a1\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.952924 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.952836 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcb321a-95e1-414c-bafc-c819681071a1-tls-certs\") pod \"kube-auth-proxy-94bcdf98f-xh2jv\" (UID: \"fbcb321a-95e1-414c-bafc-c819681071a1\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.952924 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.952885 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbcb321a-95e1-414c-bafc-c819681071a1-tmp\") pod \"kube-auth-proxy-94bcdf98f-xh2jv\" (UID: \"fbcb321a-95e1-414c-bafc-c819681071a1\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.953130 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.953003 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4hg\" (UniqueName: \"kubernetes.io/projected/fbcb321a-95e1-414c-bafc-c819681071a1-kube-api-access-2g4hg\") pod \"kube-auth-proxy-94bcdf98f-xh2jv\" (UID: \"fbcb321a-95e1-414c-bafc-c819681071a1\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.955266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.955235 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbcb321a-95e1-414c-bafc-c819681071a1-tmp\") pod \"kube-auth-proxy-94bcdf98f-xh2jv\" (UID: \"fbcb321a-95e1-414c-bafc-c819681071a1\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.955416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.955398 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcb321a-95e1-414c-bafc-c819681071a1-tls-certs\") pod \"kube-auth-proxy-94bcdf98f-xh2jv\" (UID: \"fbcb321a-95e1-414c-bafc-c819681071a1\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.976145 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.976083 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4hg\" (UniqueName: \"kubernetes.io/projected/fbcb321a-95e1-414c-bafc-c819681071a1-kube-api-access-2g4hg\") pod \"kube-auth-proxy-94bcdf98f-xh2jv\" (UID: \"fbcb321a-95e1-414c-bafc-c819681071a1\") " pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:13.977666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:13.977623 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:14.025424 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:14.025387 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" Oct 31 00:23:14.163895 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:14.163866 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:14.353461 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:14.353422 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv"] Oct 31 00:23:14.354615 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:23:14.354588 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcb321a_95e1_414c_bafc_c819681071a1.slice/crio-cc29000269a24369835409a634cac1e3771bd5a5185a4a3d64fe07092ae82265 WatchSource:0}: Error finding container cc29000269a24369835409a634cac1e3771bd5a5185a4a3d64fe07092ae82265: Status 404 returned error can't find the container with id cc29000269a24369835409a634cac1e3771bd5a5185a4a3d64fe07092ae82265 Oct 31 00:23:14.403752 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:14.403721 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:14.442091 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:14.442060 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" event={"ID":"fbcb321a-95e1-414c-bafc-c819681071a1","Type":"ContainerStarted","Data":"cc29000269a24369835409a634cac1e3771bd5a5185a4a3d64fe07092ae82265"} Oct 31 00:23:15.435679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:15.435640 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:15.927706 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:15.927672 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:16.110516 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:16.110436 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:16.413714 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:16.413677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:17.459307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.459271 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" event={"ID":"fbcb321a-95e1-414c-bafc-c819681071a1","Type":"ContainerStarted","Data":"6d81e75c71daaff1e58d5529f69be851a69d5450a3c0632cb5a613443b0afb6b"} Oct 31 00:23:17.501567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.501509 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-94bcdf98f-xh2jv" podStartSLOduration=2.075566118 podStartE2EDuration="4.50148938s" totalImagesPullingTime="2.425923262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:17.50038359 +0000 UTC m=+215.198497366" watchObservedRunningTime="2026-10-31 00:23:17.50148938 +0000 UTC m=+215.199603149" Oct 31 00:23:17.800809 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.800734 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96"] Oct 31 00:23:17.804565 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.804545 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.821354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.821325 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 00:23:17.821354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.821328 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 00:23:17.857392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.857347 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 00:23:17.857658 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.857517 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-82kzh\"" Oct 31 00:23:17.858833 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.858811 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96"] Oct 31 00:23:17.890431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.890382 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911b72ad-f153-41d5-86e2-05bcc281c020-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.890657 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.890494 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/911b72ad-f153-41d5-86e2-05bcc281c020-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.890657 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.890549 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhcd\" (UniqueName: \"kubernetes.io/projected/911b72ad-f153-41d5-86e2-05bcc281c020-kube-api-access-9lhcd\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.890790 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.890683 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/911b72ad-f153-41d5-86e2-05bcc281c020-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.991809 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.991774 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/911b72ad-f153-41d5-86e2-05bcc281c020-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.991995 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.991846 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911b72ad-f153-41d5-86e2-05bcc281c020-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.991995 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.991878 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/911b72ad-f153-41d5-86e2-05bcc281c020-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.991995 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.991911 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhcd\" (UniqueName: \"kubernetes.io/projected/911b72ad-f153-41d5-86e2-05bcc281c020-kube-api-access-9lhcd\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.992376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.992356 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911b72ad-f153-41d5-86e2-05bcc281c020-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.992423 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.992371 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/911b72ad-f153-41d5-86e2-05bcc281c020-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:17.992705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:17.992686 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/911b72ad-f153-41d5-86e2-05bcc281c020-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:18.023404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:18.023361 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhcd\" (UniqueName: \"kubernetes.io/projected/911b72ad-f153-41d5-86e2-05bcc281c020-kube-api-access-9lhcd\") pod \"kserve-module-controller-manager-7566cf5fc8-g5g96\" (UID: \"911b72ad-f153-41d5-86e2-05bcc281c020\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:18.117435 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:18.117392 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:18.445751 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:18.445721 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96"] Oct 31 00:23:18.447182 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:23:18.447156 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911b72ad_f153_41d5_86e2_05bcc281c020.slice/crio-d7bc71a3775ce757729769e6d4883b5e194e57fb5251a9c9bbcecb79ecbcc59a WatchSource:0}: Error finding container d7bc71a3775ce757729769e6d4883b5e194e57fb5251a9c9bbcecb79ecbcc59a: Status 404 returned error can't find the container with id d7bc71a3775ce757729769e6d4883b5e194e57fb5251a9c9bbcecb79ecbcc59a Oct 31 00:23:18.465304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:18.465267 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" event={"ID":"911b72ad-f153-41d5-86e2-05bcc281c020","Type":"ContainerStarted","Data":"d7bc71a3775ce757729769e6d4883b5e194e57fb5251a9c9bbcecb79ecbcc59a"} Oct 31 00:23:19.669052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:19.669015 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:19.827293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:19.827257 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:20.476893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:20.476856 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" event={"ID":"911b72ad-f153-41d5-86e2-05bcc281c020","Type":"ContainerStarted","Data":"9fe08f300de19479fb9911ddcd12b3a972e0f342153a1860a55cd76d4479c866"} Oct 31 00:23:20.477076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:20.476997 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:20.532583 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:20.532531 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" podStartSLOduration=1.924323607 podStartE2EDuration="3.532515915s" totalImagesPullingTime="1.608192308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:20.532067136 +0000 UTC m=+218.230180900" watchObservedRunningTime="2026-10-31 00:23:20.532515915 +0000 UTC m=+218.230629679" Oct 31 00:23:21.121557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:21.121517 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:21.733672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:21.733639 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:21.876296 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:21.876258 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:23.852262 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:23.852217 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:23.998305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:23.998214 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:24.520584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.520546 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9"] Oct 31 00:23:24.524164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.524140 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.548788 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.548747 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ab77c51-2344-4552-a8f1-d4312c49f8c0-cert\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.548960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.548884 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ab77c51-2344-4552-a8f1-d4312c49f8c0-metrics-certs\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.548960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.548955 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ab77c51-2344-4552-a8f1-d4312c49f8c0-manager-config\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.549080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.549051 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjb4\" (UniqueName: \"kubernetes.io/projected/4ab77c51-2344-4552-a8f1-d4312c49f8c0-kube-api-access-bzjb4\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.552006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.551957 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 00:23:24.552006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.551973 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 00:23:24.552006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.551992 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-kzghp\"" Oct 31 00:23:24.552006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.551967 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 00:23:24.553763 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.553734 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9"] Oct 31 00:23:24.649878 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.649844 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ab77c51-2344-4552-a8f1-d4312c49f8c0-metrics-certs\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.650087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.649896 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ab77c51-2344-4552-a8f1-d4312c49f8c0-manager-config\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.650087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.649933 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjb4\" (UniqueName: \"kubernetes.io/projected/4ab77c51-2344-4552-a8f1-d4312c49f8c0-kube-api-access-bzjb4\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.650087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.649960 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ab77c51-2344-4552-a8f1-d4312c49f8c0-cert\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.650795 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.650771 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ab77c51-2344-4552-a8f1-d4312c49f8c0-manager-config\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.652408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.652386 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ab77c51-2344-4552-a8f1-d4312c49f8c0-cert\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.652527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.652494 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ab77c51-2344-4552-a8f1-d4312c49f8c0-metrics-certs\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.678190 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.678149 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjb4\" (UniqueName: \"kubernetes.io/projected/4ab77c51-2344-4552-a8f1-d4312c49f8c0-kube-api-access-bzjb4\") pod \"jobset-controller-manager-85dbc6998d-gjvr9\" (UID: \"4ab77c51-2344-4552-a8f1-d4312c49f8c0\") " pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:24.836724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:24.836692 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:25.148343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:25.148307 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9"] Oct 31 00:23:25.148765 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:23:25.148672 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab77c51_2344_4552_a8f1_d4312c49f8c0.slice/crio-b80c342c537fbc7ad03f9da07989002086154d260441b250ff20f8b194def480 WatchSource:0}: Error finding container b80c342c537fbc7ad03f9da07989002086154d260441b250ff20f8b194def480: Status 404 returned error can't find the container with id b80c342c537fbc7ad03f9da07989002086154d260441b250ff20f8b194def480 Oct 31 00:23:25.500333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:25.500247 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" event={"ID":"4ab77c51-2344-4552-a8f1-d4312c49f8c0","Type":"ContainerStarted","Data":"b80c342c537fbc7ad03f9da07989002086154d260441b250ff20f8b194def480"} Oct 31 00:23:28.523886 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:28.523847 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" event={"ID":"4ab77c51-2344-4552-a8f1-d4312c49f8c0","Type":"ContainerStarted","Data":"b1e874b62c9dfd00f3062e422582e393e26f7664aff428a707ba75e3e4509d4f"} Oct 31 00:23:28.524310 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:28.523968 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:28.566971 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:28.566918 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" podStartSLOduration=1.9642295509999999 podStartE2EDuration="4.566904455s" totalImagesPullingTime="2.602674904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:28.565764307 +0000 UTC m=+226.263878072" watchObservedRunningTime="2026-10-31 00:23:28.566904455 +0000 UTC m=+226.265018284" Oct 31 00:23:28.798923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:28.798839 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:28.947786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:28.947749 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:32.447065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:32.447028 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:32.606525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:32.606492 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:33.193856 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:23:33.193810 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:23:35.411435 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:35.411395 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:35.555353 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:35.555312 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:39.534036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:39.534003 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-85dbc6998d-gjvr9" Oct 31 00:23:41.092901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:41.092862 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:41.266058 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:41.266022 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:41.482256 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:41.482172 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g5g96" Oct 31 00:23:41.835940 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:23:41.835903 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:03.181402 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:24:03.181358 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:24:16.645074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:16.645039 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:16.869348 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:16.869310 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:17.130246 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:17.130209 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:17.519736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:17.519644 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:17.799790 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:17.799697 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:19.827537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:19.827496 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:20.006655 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:20.006620 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:20.198711 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:20.198627 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:20.367211 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:20.367167 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:33.180202 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:24:33.180165 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:24:41.884767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:41.884729 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:42.466759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:42.466728 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:24:42.467827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:24:42.467804 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:25:03.179856 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:25:03.179813 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:25:17.320869 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:25:17.320830 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:17.527135 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:25:17.527101 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:33.180080 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:25:33.180042 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:25:42.017003 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:25:42.016958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:03.179870 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:26:03.179787 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:26:33.181075 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:26:33.181034 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:26:42.133558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:26:42.133522 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:03.180374 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:27:03.180332 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:27:33.181936 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:27:33.181846 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:27:42.252607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:27:42.252571 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:03.181029 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:28:03.180989 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:28:31.436843 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:28:31.436804 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:33.180904 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:28:33.180867 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:28:42.396914 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:28:42.396874 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:03.181574 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:29:03.181476 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:29:16.152078 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:29:16.152039 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:33.179832 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:29:33.179792 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:29:42.503234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:29:42.503204 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:29:42.504235 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:29:42.504210 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:29:42.522779 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:29:42.522746 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:03.179847 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:30:03.179805 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:30:05.454527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:30:05.454488 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:33.180174 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:30:33.180088 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:30:42.603147 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:30:42.603097 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.002496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:30:47.002462 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.167782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:30:47.167744 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:03.180111 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:31:03.180070 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:31:30.185851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:31:30.185812 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:33.179911 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:31:33.179873 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:31:42.756789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:31:42.756742 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:43.005962 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:31:43.005921 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:03.182550 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:32:03.182443 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:32:33.181300 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:32:33.181260 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:32:42.887389 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:32:42.887349 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:03.180409 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:33:03.180368 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:33:33.180975 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:33:33.180869 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:33:39.312629 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:39.312591 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:43.026536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:43.026489 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:51.800164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:51.800126 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:51.953308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:51.953266 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:52.120526 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.120493 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:52.686392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.686354 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-k858c"] Oct 31 00:33:52.690429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.690404 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-k858c" Oct 31 00:33:52.712954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.712917 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t5h5b\"" Oct 31 00:33:52.724286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.724252 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-k858c"] Oct 31 00:33:52.773983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.773946 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:33:52.773983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.773981 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:33:52.791465 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.791412 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42jl\" (UniqueName: \"kubernetes.io/projected/20bb18cb-4d74-4316-8bac-a8e32b7d254c-kube-api-access-r42jl\") pod \"authorino-57f7f8659f-k858c\" (UID: \"20bb18cb-4d74-4316-8bac-a8e32b7d254c\") " pod="kuadrant-system/authorino-57f7f8659f-k858c" Oct 31 00:33:52.821801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.821763 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:52.892575 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.892534 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42jl\" (UniqueName: \"kubernetes.io/projected/20bb18cb-4d74-4316-8bac-a8e32b7d254c-kube-api-access-r42jl\") pod \"authorino-57f7f8659f-k858c\" (UID: \"20bb18cb-4d74-4316-8bac-a8e32b7d254c\") " pod="kuadrant-system/authorino-57f7f8659f-k858c" Oct 31 00:33:52.914555 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.914524 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42jl\" (UniqueName: \"kubernetes.io/projected/20bb18cb-4d74-4316-8bac-a8e32b7d254c-kube-api-access-r42jl\") pod \"authorino-57f7f8659f-k858c\" (UID: \"20bb18cb-4d74-4316-8bac-a8e32b7d254c\") " pod="kuadrant-system/authorino-57f7f8659f-k858c" Oct 31 00:33:52.975931 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:52.975846 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:53.002613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:53.002575 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-k858c" Oct 31 00:33:53.293720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:53.293626 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-k858c"] Oct 31 00:33:53.296868 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:33:53.296837 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20bb18cb_4d74_4316_8bac_a8e32b7d254c.slice/crio-26b658b21626555f60115641e22cadd5a07d286fd38166b7f19b82f2400132a0 WatchSource:0}: Error finding container 26b658b21626555f60115641e22cadd5a07d286fd38166b7f19b82f2400132a0: Status 404 returned error can't find the container with id 26b658b21626555f60115641e22cadd5a07d286fd38166b7f19b82f2400132a0 Oct 31 00:33:53.298551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:53.298528 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:33:53.409527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:53.409494 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-k858c" event={"ID":"20bb18cb-4d74-4316-8bac-a8e32b7d254c","Type":"ContainerStarted","Data":"26b658b21626555f60115641e22cadd5a07d286fd38166b7f19b82f2400132a0"} Oct 31 00:33:56.410554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:56.410509 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-k858c"] Oct 31 00:33:56.426184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:56.426147 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-k858c" event={"ID":"20bb18cb-4d74-4316-8bac-a8e32b7d254c","Type":"ContainerStarted","Data":"151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca"} Oct 31 00:33:56.463835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:56.463775 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-k858c" podStartSLOduration=2.011513283 podStartE2EDuration="4.46375265s" totalImagesPullingTime="2.452239367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:56.460542914 +0000 UTC m=+854.158656679" watchObservedRunningTime="2026-10-31 00:33:56.46375265 +0000 UTC m=+854.161866417" Oct 31 00:33:57.430857 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:57.430802 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-k858c" podUID="20bb18cb-4d74-4316-8bac-a8e32b7d254c" containerName="authorino" containerID="cri-o://151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca" gracePeriod=30 Oct 31 00:33:57.687590 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:57.687530 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-k858c" Oct 31 00:33:57.839106 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:57.839067 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bb18cb-4d74-4316-8bac-a8e32b7d254c-kube-api-access-r42jl\" (UniqueName: \"kubernetes.io/projected/20bb18cb-4d74-4316-8bac-a8e32b7d254c-kube-api-access-r42jl\") pod \"20bb18cb-4d74-4316-8bac-a8e32b7d254c\" (UID: \"20bb18cb-4d74-4316-8bac-a8e32b7d254c\") " Oct 31 00:33:57.841210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:57.841187 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bb18cb-4d74-4316-8bac-a8e32b7d254c-kube-api-access-r42jl" pod "20bb18cb-4d74-4316-8bac-a8e32b7d254c" (UID: "20bb18cb-4d74-4316-8bac-a8e32b7d254c"). InnerVolumeSpecName "kube-api-access-r42jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:57.940324 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:57.940233 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r42jl\" (UniqueName: \"kubernetes.io/projected/20bb18cb-4d74-4316-8bac-a8e32b7d254c-kube-api-access-r42jl\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:33:58.436176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:58.436148 5978 generic.go:356] "Generic (PLEG): container finished" podID="20bb18cb-4d74-4316-8bac-a8e32b7d254c" containerID="151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca" exitCode=0 Oct 31 00:33:58.436607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:58.436186 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-k858c" Oct 31 00:33:58.436607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:58.436227 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-k858c" event={"ID":"20bb18cb-4d74-4316-8bac-a8e32b7d254c","Type":"ContainerDied","Data":"151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca"} Oct 31 00:33:58.436607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:58.436265 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-k858c" event={"ID":"20bb18cb-4d74-4316-8bac-a8e32b7d254c","Type":"ContainerDied","Data":"26b658b21626555f60115641e22cadd5a07d286fd38166b7f19b82f2400132a0"} Oct 31 00:33:58.436607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:58.436281 5978 scope.go:122] "RemoveContainer" containerID="151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca" Oct 31 00:33:58.444705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:58.444687 5978 scope.go:122] "RemoveContainer" containerID="151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca" Oct 31 00:33:58.444944 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:33:58.444921 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca\": container with ID starting with 151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca not found: ID does not exist" containerID="151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca" Oct 31 00:33:58.444994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:58.444949 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca"} err="failed to get container status \"151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca\": rpc error: code = NotFound desc = could not find container \"151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca\": container with ID starting with 151b3d2793756be2f8baaf47766e2ca8c1539a9a7cdddae119947aed315160ca not found: ID does not exist" Oct 31 00:33:58.480178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:58.480145 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-k858c"] Oct 31 00:33:58.490036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:33:58.490004 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-k858c"] Oct 31 00:34:00.423550 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:00.423517 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20bb18cb-4d74-4316-8bac-a8e32b7d254c" path="/var/lib/kubelet/pods/20bb18cb-4d74-4316-8bac-a8e32b7d254c/volumes" Oct 31 00:34:03.181525 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:34:03.181486 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:34:06.453273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:06.453236 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:06.595740 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:06.595698 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:08.939556 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:08.939527 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:09.109664 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:09.109635 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:09.714265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:09.714225 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:28.678547 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:28.678495 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:28.821616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:28.821585 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:28.967634 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:28.967554 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:32.604099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:32.604055 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:32.763998 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:32.763963 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:33.180892 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:34:33.180851 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:34:34.555963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.555931 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g"] Oct 31 00:34:34.556421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.556310 5978 cpu_manager.go:414] "containerMap: removing container" podUID="20bb18cb-4d74-4316-8bac-a8e32b7d254c" containerName="authorino" Oct 31 00:34:34.556421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.556321 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20bb18cb-4d74-4316-8bac-a8e32b7d254c" containerName="authorino" Oct 31 00:34:34.556421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.556402 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="20bb18cb-4d74-4316-8bac-a8e32b7d254c" containerName="authorino" Oct 31 00:34:34.561038 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.561013 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.575999 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.575967 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 00:34:34.585941 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.585907 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qmd9b\"" Oct 31 00:34:34.586129 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.586007 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 00:34:34.586827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.586694 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 00:34:34.587654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.587627 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g"] Oct 31 00:34:34.671756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.671721 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:34.685915 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.685881 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/097ad6a1-77bb-405a-8fb5-838805f787c4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.685915 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.685914 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86f59\" (UniqueName: \"kubernetes.io/projected/097ad6a1-77bb-405a-8fb5-838805f787c4-kube-api-access-86f59\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.686185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.686012 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/097ad6a1-77bb-405a-8fb5-838805f787c4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.686185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.686094 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/097ad6a1-77bb-405a-8fb5-838805f787c4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.786924 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.786887 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/097ad6a1-77bb-405a-8fb5-838805f787c4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.787141 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.786933 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86f59\" (UniqueName: \"kubernetes.io/projected/097ad6a1-77bb-405a-8fb5-838805f787c4-kube-api-access-86f59\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.787141 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.786993 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/097ad6a1-77bb-405a-8fb5-838805f787c4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.787141 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.787020 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/097ad6a1-77bb-405a-8fb5-838805f787c4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.787757 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.787731 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/097ad6a1-77bb-405a-8fb5-838805f787c4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.787984 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.787965 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/097ad6a1-77bb-405a-8fb5-838805f787c4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.789396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.789375 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/097ad6a1-77bb-405a-8fb5-838805f787c4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.808381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.808304 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86f59\" (UniqueName: \"kubernetes.io/projected/097ad6a1-77bb-405a-8fb5-838805f787c4-kube-api-access-86f59\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6tv7g\" (UID: \"097ad6a1-77bb-405a-8fb5-838805f787c4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.873618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.873575 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:34.885231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:34.885190 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:35.166982 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:35.166948 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g"] Oct 31 00:34:35.170567 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:34:35.170532 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097ad6a1_77bb_405a_8fb5_838805f787c4.slice/crio-77e042253f8223a31e1d5e70decab453dfc5d786e3ac38356dfb062ce872d6ae WatchSource:0}: Error finding container 77e042253f8223a31e1d5e70decab453dfc5d786e3ac38356dfb062ce872d6ae: Status 404 returned error can't find the container with id 77e042253f8223a31e1d5e70decab453dfc5d786e3ac38356dfb062ce872d6ae Oct 31 00:34:35.608903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:35.608865 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" event={"ID":"097ad6a1-77bb-405a-8fb5-838805f787c4","Type":"ContainerStarted","Data":"77e042253f8223a31e1d5e70decab453dfc5d786e3ac38356dfb062ce872d6ae"} Oct 31 00:34:42.546839 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:42.546814 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:34:42.549083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:42.549061 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:34:43.146361 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:43.146320 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:51.696896 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:51.696855 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" event={"ID":"097ad6a1-77bb-405a-8fb5-838805f787c4","Type":"ContainerStarted","Data":"c027d8fad0fefbca9a4a6dd7a53408f585bcb04d8aa4d87a076e1f1781c4f135"} Oct 31 00:34:51.697304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:51.696965 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:34:51.736200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:34:51.736145 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" podStartSLOduration=1.860122083 podStartE2EDuration="17.736131794s" totalImagesPullingTime="15.876009711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:51.734793229 +0000 UTC m=+909.432906996" watchObservedRunningTime="2026-10-31 00:34:51.736131794 +0000 UTC m=+909.434245559" Oct 31 00:35:02.707317 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:02.707240 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6tv7g" Oct 31 00:35:03.179865 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:35:03.179825 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:35:04.926552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:04.926504 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:05.079149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:05.079109 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:06.633435 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:06.633403 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:06.780141 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:06.780105 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:16.187871 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.187836 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:16.454672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.454582 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9dw86"] Oct 31 00:35:16.470992 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.470962 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9dw86" Oct 31 00:35:16.474757 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.474730 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9dw86"] Oct 31 00:35:16.483362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.483332 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:35:16.483544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.483361 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:35:16.493266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.493237 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t5h5b\"" Oct 31 00:35:16.570642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.570604 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqtmc\" (UniqueName: \"kubernetes.io/projected/1d8b4d55-a182-475b-a49d-f86991690002-kube-api-access-vqtmc\") pod \"authorino-6f6944bcdd-9dw86\" (UID: \"1d8b4d55-a182-475b-a49d-f86991690002\") " pod="kuadrant-system/authorino-6f6944bcdd-9dw86" Oct 31 00:35:16.651725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.651691 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9dw86"] Oct 31 00:35:16.651929 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:35:16.651908 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vqtmc], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-9dw86" podUID="1d8b4d55-a182-475b-a49d-f86991690002" Oct 31 00:35:16.671938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.671905 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqtmc\" (UniqueName: \"kubernetes.io/projected/1d8b4d55-a182-475b-a49d-f86991690002-kube-api-access-vqtmc\") pod \"authorino-6f6944bcdd-9dw86\" (UID: \"1d8b4d55-a182-475b-a49d-f86991690002\") " pod="kuadrant-system/authorino-6f6944bcdd-9dw86" Oct 31 00:35:16.700216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.700188 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqtmc\" (UniqueName: \"kubernetes.io/projected/1d8b4d55-a182-475b-a49d-f86991690002-kube-api-access-vqtmc\") pod \"authorino-6f6944bcdd-9dw86\" (UID: \"1d8b4d55-a182-475b-a49d-f86991690002\") " pod="kuadrant-system/authorino-6f6944bcdd-9dw86" Oct 31 00:35:16.814650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.814574 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9dw86" Oct 31 00:35:16.848158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.848126 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54d45fccc5-xwt25"] Oct 31 00:35:16.877958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.877917 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9dw86" Oct 31 00:35:16.893621 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.893588 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54d45fccc5-xwt25"] Oct 31 00:35:16.893770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.893675 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:35:16.906106 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.906083 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 00:35:16.974750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.974711 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8b4d55-a182-475b-a49d-f86991690002-kube-api-access-vqtmc\" (UniqueName: \"kubernetes.io/projected/1d8b4d55-a182-475b-a49d-f86991690002-kube-api-access-vqtmc\") pod \"1d8b4d55-a182-475b-a49d-f86991690002\" (UID: \"1d8b4d55-a182-475b-a49d-f86991690002\") " Oct 31 00:35:16.974944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.974868 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1729574-6142-4180-a046-a93ccd733932-tls-cert\") pod \"authorino-54d45fccc5-xwt25\" (UID: \"b1729574-6142-4180-a046-a93ccd733932\") " pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:35:16.974944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.974917 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnzm\" (UniqueName: \"kubernetes.io/projected/b1729574-6142-4180-a046-a93ccd733932-kube-api-access-qsnzm\") pod \"authorino-54d45fccc5-xwt25\" (UID: \"b1729574-6142-4180-a046-a93ccd733932\") " pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:35:16.976796 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:16.976768 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8b4d55-a182-475b-a49d-f86991690002-kube-api-access-vqtmc" pod "1d8b4d55-a182-475b-a49d-f86991690002" (UID: "1d8b4d55-a182-475b-a49d-f86991690002"). InnerVolumeSpecName "kube-api-access-vqtmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:17.075769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.075732 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1729574-6142-4180-a046-a93ccd733932-tls-cert\") pod \"authorino-54d45fccc5-xwt25\" (UID: \"b1729574-6142-4180-a046-a93ccd733932\") " pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:35:17.075937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.075786 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnzm\" (UniqueName: \"kubernetes.io/projected/b1729574-6142-4180-a046-a93ccd733932-kube-api-access-qsnzm\") pod \"authorino-54d45fccc5-xwt25\" (UID: \"b1729574-6142-4180-a046-a93ccd733932\") " pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:35:17.075937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.075823 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqtmc\" (UniqueName: \"kubernetes.io/projected/1d8b4d55-a182-475b-a49d-f86991690002-kube-api-access-vqtmc\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:35:17.078291 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.078267 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1729574-6142-4180-a046-a93ccd733932-tls-cert\") pod \"authorino-54d45fccc5-xwt25\" (UID: \"b1729574-6142-4180-a046-a93ccd733932\") " pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:35:17.097156 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.097127 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnzm\" (UniqueName: \"kubernetes.io/projected/b1729574-6142-4180-a046-a93ccd733932-kube-api-access-qsnzm\") pod \"authorino-54d45fccc5-xwt25\" (UID: \"b1729574-6142-4180-a046-a93ccd733932\") " pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:35:17.237795 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.237757 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:35:17.557622 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.557516 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54d45fccc5-xwt25"] Oct 31 00:35:17.560872 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:35:17.560845 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1729574_6142_4180_a046_a93ccd733932.slice/crio-16b733ebefb328947cf2aee90e03d3f31315c5467019ebbf756a86e66cc1921b WatchSource:0}: Error finding container 16b733ebefb328947cf2aee90e03d3f31315c5467019ebbf756a86e66cc1921b: Status 404 returned error can't find the container with id 16b733ebefb328947cf2aee90e03d3f31315c5467019ebbf756a86e66cc1921b Oct 31 00:35:17.714186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.714144 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:17.821207 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.821173 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d45fccc5-xwt25" event={"ID":"b1729574-6142-4180-a046-a93ccd733932","Type":"ContainerStarted","Data":"16b733ebefb328947cf2aee90e03d3f31315c5467019ebbf756a86e66cc1921b"} Oct 31 00:35:17.821358 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.821220 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9dw86" Oct 31 00:35:17.888096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.888058 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9dw86"] Oct 31 00:35:17.891739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:17.891710 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9dw86"] Oct 31 00:35:18.423580 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:18.423545 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8b4d55-a182-475b-a49d-f86991690002" path="/var/lib/kubelet/pods/1d8b4d55-a182-475b-a49d-f86991690002/volumes" Oct 31 00:35:26.863898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:26.863858 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d45fccc5-xwt25" event={"ID":"b1729574-6142-4180-a046-a93ccd733932","Type":"ContainerStarted","Data":"b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb"} Oct 31 00:35:26.901959 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:26.901905 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-54d45fccc5-xwt25" podStartSLOduration=2.306460655 podStartE2EDuration="10.901893366s" totalImagesPullingTime="8.595432711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:26.900773014 +0000 UTC m=+944.598886778" watchObservedRunningTime="2026-10-31 00:35:26.901893366 +0000 UTC m=+944.600007128" Oct 31 00:35:33.180104 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:35:33.180063 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:35:43.315694 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:35:43.315655 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:01.584862 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:01.584825 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:01.735203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:01.735162 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:03.179715 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:36:03.179681 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:36:19.762352 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:19.762305 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:21.924322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:21.924284 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:24.014233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.014144 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9"] Oct 31 00:36:24.017009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.016987 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.042281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.042242 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 00:36:24.042582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.042278 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gdwmn\"" Oct 31 00:36:24.042700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.042407 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 00:36:24.042818 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.042784 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 00:36:24.044268 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.044240 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9"] Oct 31 00:36:24.085753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.085702 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847df7f7-0054-4cc9-a296-e64c9f10dd2d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.085753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.085754 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.086006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.085846 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hrj\" (UniqueName: \"kubernetes.io/projected/847df7f7-0054-4cc9-a296-e64c9f10dd2d-kube-api-access-w6hrj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.086006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.085885 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.086006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.085981 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.086180 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.086016 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.141534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.141495 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:24.186608 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.186569 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.186803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.186633 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.186803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.186673 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.186803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.186723 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847df7f7-0054-4cc9-a296-e64c9f10dd2d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.186803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.186750 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.187074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.186822 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hrj\" (UniqueName: \"kubernetes.io/projected/847df7f7-0054-4cc9-a296-e64c9f10dd2d-kube-api-access-w6hrj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.187074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.187037 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.187351 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.187092 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.187351 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.187144 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.189104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.189073 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847df7f7-0054-4cc9-a296-e64c9f10dd2d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.189329 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.189313 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847df7f7-0054-4cc9-a296-e64c9f10dd2d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.208201 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.208173 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hrj\" (UniqueName: \"kubernetes.io/projected/847df7f7-0054-4cc9-a296-e64c9f10dd2d-kube-api-access-w6hrj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9\" (UID: \"847df7f7-0054-4cc9-a296-e64c9f10dd2d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.331908 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.331866 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:24.628095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:24.627946 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9"] Oct 31 00:36:24.630483 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:36:24.630436 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod847df7f7_0054_4cc9_a296_e64c9f10dd2d.slice/crio-7429fcac499f03b6b7d98ea9a273bdcf4c59ab6237cd96a8a4cb2f1b3915b97d WatchSource:0}: Error finding container 7429fcac499f03b6b7d98ea9a273bdcf4c59ab6237cd96a8a4cb2f1b3915b97d: Status 404 returned error can't find the container with id 7429fcac499f03b6b7d98ea9a273bdcf4c59ab6237cd96a8a4cb2f1b3915b97d Oct 31 00:36:25.134537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:25.134496 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" event={"ID":"847df7f7-0054-4cc9-a296-e64c9f10dd2d","Type":"ContainerStarted","Data":"7429fcac499f03b6b7d98ea9a273bdcf4c59ab6237cd96a8a4cb2f1b3915b97d"} Oct 31 00:36:25.451631 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:25.451523 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:26.141196 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:26.141150 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" event={"ID":"847df7f7-0054-4cc9-a296-e64c9f10dd2d","Type":"ContainerStarted","Data":"7b0c1ab0a438ef7160b6cb1b6710ecd215875c208d7be2cc965e653eafe92cb5"} Oct 31 00:36:26.181955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:26.181898 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podStartSLOduration=2.069868509 podStartE2EDuration="3.181880794s" totalImagesPullingTime="1.112012285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:26.18082189 +0000 UTC m=+1003.878935655" watchObservedRunningTime="2026-10-31 00:36:26.181880794 +0000 UTC m=+1003.879994536" Oct 31 00:36:26.332312 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:26.332272 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:27.094630 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.094593 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj"] Oct 31 00:36:27.099967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.099941 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.113474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.113397 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.113474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.113438 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51fb742e-ddac-4178-8f67-38f62e8d3f71-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.113765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.113595 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wkpj\" (UniqueName: \"kubernetes.io/projected/51fb742e-ddac-4178-8f67-38f62e8d3f71-kube-api-access-7wkpj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.113765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.113678 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.113765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.113735 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.113957 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.113776 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.122550 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.122524 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 00:36:27.124265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.124243 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj"] Oct 31 00:36:27.214367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.214328 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wkpj\" (UniqueName: \"kubernetes.io/projected/51fb742e-ddac-4178-8f67-38f62e8d3f71-kube-api-access-7wkpj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.214367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.214368 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.214989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.214413 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.214989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.214444 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.214989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.214549 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.214989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.214571 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51fb742e-ddac-4178-8f67-38f62e8d3f71-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.214989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.214903 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.214989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.214928 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.214989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.214968 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.216689 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.216667 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51fb742e-ddac-4178-8f67-38f62e8d3f71-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.217063 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.217043 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51fb742e-ddac-4178-8f67-38f62e8d3f71-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.262734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.262704 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wkpj\" (UniqueName: \"kubernetes.io/projected/51fb742e-ddac-4178-8f67-38f62e8d3f71-kube-api-access-7wkpj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cckrj\" (UID: \"51fb742e-ddac-4178-8f67-38f62e8d3f71\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.300385 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.300354 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:27.414882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.414792 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:27.708567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:27.708479 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj"] Oct 31 00:36:27.711394 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:36:27.711370 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fb742e_ddac_4178_8f67_38f62e8d3f71.slice/crio-c19884244982c099242749247ae7b8e696f9ba58657af48160dc2a3ddcec3108 WatchSource:0}: Error finding container c19884244982c099242749247ae7b8e696f9ba58657af48160dc2a3ddcec3108: Status 404 returned error can't find the container with id c19884244982c099242749247ae7b8e696f9ba58657af48160dc2a3ddcec3108 Oct 31 00:36:28.154054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:28.154016 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" event={"ID":"51fb742e-ddac-4178-8f67-38f62e8d3f71","Type":"ContainerStarted","Data":"17f3e1c036a6e59a74e3f4f5b94d9ed71408f1a6659a7028de837e39ed42f448"} Oct 31 00:36:28.154221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:28.154074 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" event={"ID":"51fb742e-ddac-4178-8f67-38f62e8d3f71","Type":"ContainerStarted","Data":"c19884244982c099242749247ae7b8e696f9ba58657af48160dc2a3ddcec3108"} Oct 31 00:36:28.194107 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:28.194045 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" podStartSLOduration=0.9354344 podStartE2EDuration="1.194026457s" totalImagesPullingTime="258.592057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:28.193769898 +0000 UTC m=+1005.891883664" watchObservedRunningTime="2026-10-31 00:36:28.194026457 +0000 UTC m=+1005.892140222" Oct 31 00:36:28.415407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:28.415352 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:29.893198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.893158 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5"] Oct 31 00:36:29.897121 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.897097 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:29.909982 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.909954 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 00:36:29.921295 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.921270 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5"] Oct 31 00:36:29.941215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.941169 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:29.941406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.941239 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:29.941406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.941339 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:29.941406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.941386 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcb20e6-3314-47bb-983e-40d23e003f6d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:29.941612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.941512 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:29.941612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:29.941557 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkqhh\" (UniqueName: \"kubernetes.io/projected/0dcb20e6-3314-47bb-983e-40d23e003f6d-kube-api-access-rkqhh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.005883 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.005849 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:30.043066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.043024 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkqhh\" (UniqueName: \"kubernetes.io/projected/0dcb20e6-3314-47bb-983e-40d23e003f6d-kube-api-access-rkqhh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.043271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.043188 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.043271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.043248 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.043400 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.043315 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.043400 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.043368 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcb20e6-3314-47bb-983e-40d23e003f6d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.043541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.043445 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.043708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.043687 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.043773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.043727 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.043828 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.043806 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.045685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.045656 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcb20e6-3314-47bb-983e-40d23e003f6d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.045941 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.045924 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcb20e6-3314-47bb-983e-40d23e003f6d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.064536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.064508 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkqhh\" (UniqueName: \"kubernetes.io/projected/0dcb20e6-3314-47bb-983e-40d23e003f6d-kube-api-access-rkqhh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kl9w5\" (UID: \"0dcb20e6-3314-47bb-983e-40d23e003f6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.214878 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.214793 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:30.511088 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:30.510833 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5"] Oct 31 00:36:30.512911 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:36:30.512881 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcb20e6_3314_47bb_983e_40d23e003f6d.slice/crio-aeb397dea4d720371ab5900018c707a48b38510862527a92bc1944c3398519bf WatchSource:0}: Error finding container aeb397dea4d720371ab5900018c707a48b38510862527a92bc1944c3398519bf: Status 404 returned error can't find the container with id aeb397dea4d720371ab5900018c707a48b38510862527a92bc1944c3398519bf Oct 31 00:36:31.170933 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:31.170888 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" event={"ID":"0dcb20e6-3314-47bb-983e-40d23e003f6d","Type":"ContainerStarted","Data":"49dc2ee34877f07a6344a7a747b099053b96a85f0e079b91fe2d75307afb0950"} Oct 31 00:36:31.171347 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:31.170960 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" event={"ID":"0dcb20e6-3314-47bb-983e-40d23e003f6d","Type":"ContainerStarted","Data":"aeb397dea4d720371ab5900018c707a48b38510862527a92bc1944c3398519bf"} Oct 31 00:36:31.209759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:31.209704 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" podStartSLOduration=2.000916082 podStartE2EDuration="2.209688179s" totalImagesPullingTime="208.772097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:31.208935581 +0000 UTC m=+1008.907049349" watchObservedRunningTime="2026-10-31 00:36:31.209688179 +0000 UTC m=+1008.907801944" Oct 31 00:36:31.215012 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:31.214985 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:31.650375 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:31.650341 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:33.181927 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:36:33.181877 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:36:33.908933 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:33.908895 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:34.059994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:34.059958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:34.332307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:34.332279 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:34.344879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:34.344842 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:35.205434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:35.205405 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:36:37.415136 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:37.415091 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:37.429773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:37.429740 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:38.221512 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:38.221477 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cckrj" Oct 31 00:36:40.215717 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:40.215684 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:40.228609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:40.228584 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:41.227965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.227935 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" Oct 31 00:36:41.844014 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.843978 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg"] Oct 31 00:36:41.848004 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.847980 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:41.860294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.860267 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 00:36:41.860487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.860475 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vtxt8\"" Oct 31 00:36:41.877298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.877258 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg"] Oct 31 00:36:41.970379 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.970343 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:41.970379 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.970383 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:41.970618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.970404 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8a644-19f4-4af9-84bb-964324d42272-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:41.970618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.970422 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:41.970618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.970572 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:41.970714 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.970616 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqdhf\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-kube-api-access-tqdhf\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:41.970748 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.970721 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:41.970781 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.970749 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:41.970853 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:41.970841 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8a644-19f4-4af9-84bb-964324d42272-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072140 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072103 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8a644-19f4-4af9-84bb-964324d42272-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072140 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072145 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072164 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072183 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8a644-19f4-4af9-84bb-964324d42272-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072327 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072378 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072422 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqdhf\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-kube-api-access-tqdhf\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072523 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072554 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072627 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072881 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072861 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.072965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.072861 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.073085 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.073069 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.073155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.073135 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8a644-19f4-4af9-84bb-964324d42272-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.074610 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.074590 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.074718 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.074700 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8a644-19f4-4af9-84bb-964324d42272-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.094085 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.094005 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.094085 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.094070 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqdhf\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-kube-api-access-tqdhf\") pod \"maas-default-gateway-openshift-default-6886b96f9-vd2tg\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.161588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.161544 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:42.474986 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:42.474950 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg"] Oct 31 00:36:42.477056 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:36:42.477025 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a8a644_19f4_4af9_84bb_964324d42272.slice/crio-af14ad0240c38bab33b7eadb6c68f2f0a1ecf17c30fb1f9d60a66b82b31b2df1 WatchSource:0}: Error finding container af14ad0240c38bab33b7eadb6c68f2f0a1ecf17c30fb1f9d60a66b82b31b2df1: Status 404 returned error can't find the container with id af14ad0240c38bab33b7eadb6c68f2f0a1ecf17c30fb1f9d60a66b82b31b2df1 Oct 31 00:36:43.238509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:43.238472 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" event={"ID":"66a8a644-19f4-4af9-84bb-964324d42272","Type":"ContainerStarted","Data":"af14ad0240c38bab33b7eadb6c68f2f0a1ecf17c30fb1f9d60a66b82b31b2df1"} Oct 31 00:36:43.408786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:43.408748 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:45.693848 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:45.693806 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:36:45.694163 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:45.693879 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:36:45.694163 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:45.693912 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:36:46.257917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:46.257878 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" event={"ID":"66a8a644-19f4-4af9-84bb-964324d42272","Type":"ContainerStarted","Data":"3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a"} Oct 31 00:36:46.308982 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:46.308930 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podStartSLOduration=2.101692473 podStartE2EDuration="5.308915436s" totalImagesPullingTime="3.207222963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:46.30838166 +0000 UTC m=+1024.006495585" watchObservedRunningTime="2026-10-31 00:36:46.308915436 +0000 UTC m=+1024.007029203" Oct 31 00:36:47.162325 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:47.162278 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:47.163835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:47.163806 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vd2tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 00:36:47.163963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:47.163900 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 00:36:48.162900 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:48.162859 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vd2tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 00:36:48.163319 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:48.162932 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 00:36:48.895653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:48.895609 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c6f985896-klgtr"] Oct 31 00:36:48.899625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:48.899598 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c6f985896-klgtr" Oct 31 00:36:48.929478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:48.929113 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c6f985896-klgtr"] Oct 31 00:36:49.057429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.057389 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl682\" (UniqueName: \"kubernetes.io/projected/ee5bd20d-24df-423b-928c-872d211c46bd-kube-api-access-dl682\") pod \"authorino-c6f985896-klgtr\" (UID: \"ee5bd20d-24df-423b-928c-872d211c46bd\") " pod="kuadrant-system/authorino-c6f985896-klgtr" Oct 31 00:36:49.057612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.057480 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee5bd20d-24df-423b-928c-872d211c46bd-tls-cert\") pod \"authorino-c6f985896-klgtr\" (UID: \"ee5bd20d-24df-423b-928c-872d211c46bd\") " pod="kuadrant-system/authorino-c6f985896-klgtr" Oct 31 00:36:49.158532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.158424 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl682\" (UniqueName: \"kubernetes.io/projected/ee5bd20d-24df-423b-928c-872d211c46bd-kube-api-access-dl682\") pod \"authorino-c6f985896-klgtr\" (UID: \"ee5bd20d-24df-423b-928c-872d211c46bd\") " pod="kuadrant-system/authorino-c6f985896-klgtr" Oct 31 00:36:49.158532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.158528 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee5bd20d-24df-423b-928c-872d211c46bd-tls-cert\") pod \"authorino-c6f985896-klgtr\" (UID: \"ee5bd20d-24df-423b-928c-872d211c46bd\") " pod="kuadrant-system/authorino-c6f985896-klgtr" Oct 31 00:36:49.161177 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.161151 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee5bd20d-24df-423b-928c-872d211c46bd-tls-cert\") pod \"authorino-c6f985896-klgtr\" (UID: \"ee5bd20d-24df-423b-928c-872d211c46bd\") " pod="kuadrant-system/authorino-c6f985896-klgtr" Oct 31 00:36:49.162013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.161985 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vd2tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 00:36:49.162128 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.162042 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 00:36:49.179858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.179820 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl682\" (UniqueName: \"kubernetes.io/projected/ee5bd20d-24df-423b-928c-872d211c46bd-kube-api-access-dl682\") pod \"authorino-c6f985896-klgtr\" (UID: \"ee5bd20d-24df-423b-928c-872d211c46bd\") " pod="kuadrant-system/authorino-c6f985896-klgtr" Oct 31 00:36:49.228660 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.228619 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c6f985896-klgtr" Oct 31 00:36:49.534536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:49.533751 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c6f985896-klgtr"] Oct 31 00:36:49.540521 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:36:49.540488 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee5bd20d_24df_423b_928c_872d211c46bd.slice/crio-c06feff9f5248ec1c5f77e0ffeb40e28235d78b2ad2fd5d495dc9af9d61ca6f9 WatchSource:0}: Error finding container c06feff9f5248ec1c5f77e0ffeb40e28235d78b2ad2fd5d495dc9af9d61ca6f9: Status 404 returned error can't find the container with id c06feff9f5248ec1c5f77e0ffeb40e28235d78b2ad2fd5d495dc9af9d61ca6f9 Oct 31 00:36:50.162421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.162366 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vd2tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 00:36:50.162506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.162425 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 00:36:50.279964 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.279930 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c6f985896-klgtr" event={"ID":"ee5bd20d-24df-423b-928c-872d211c46bd","Type":"ContainerStarted","Data":"379e3a04584277283bf92d0f21e0ec4372a52b2e8c1f569c22e81c97835a1767"} Oct 31 00:36:50.280325 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.279976 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c6f985896-klgtr" event={"ID":"ee5bd20d-24df-423b-928c-872d211c46bd","Type":"ContainerStarted","Data":"c06feff9f5248ec1c5f77e0ffeb40e28235d78b2ad2fd5d495dc9af9d61ca6f9"} Oct 31 00:36:50.342437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.342368 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c6f985896-klgtr" podStartSLOduration=1.722340126 podStartE2EDuration="2.342349195s" totalImagesPullingTime="620.009069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:50.341758857 +0000 UTC m=+1028.039872637" watchObservedRunningTime="2026-10-31 00:36:50.342349195 +0000 UTC m=+1028.040462960" Oct 31 00:36:50.345532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.345495 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54d45fccc5-xwt25"] Oct 31 00:36:50.346148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.345789 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-54d45fccc5-xwt25" podUID="b1729574-6142-4180-a046-a93ccd733932" containerName="authorino" containerID="cri-o://b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb" gracePeriod=30 Oct 31 00:36:50.640884 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.640851 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:36:50.777209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.777103 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1729574-6142-4180-a046-a93ccd733932-kube-api-access-qsnzm\" (UniqueName: \"kubernetes.io/projected/b1729574-6142-4180-a046-a93ccd733932-kube-api-access-qsnzm\") pod \"b1729574-6142-4180-a046-a93ccd733932\" (UID: \"b1729574-6142-4180-a046-a93ccd733932\") " Oct 31 00:36:50.777522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.777213 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1729574-6142-4180-a046-a93ccd733932-tls-cert\" (UniqueName: \"kubernetes.io/secret/b1729574-6142-4180-a046-a93ccd733932-tls-cert\") pod \"b1729574-6142-4180-a046-a93ccd733932\" (UID: \"b1729574-6142-4180-a046-a93ccd733932\") " Oct 31 00:36:50.779394 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.779358 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1729574-6142-4180-a046-a93ccd733932-kube-api-access-qsnzm" pod "b1729574-6142-4180-a046-a93ccd733932" (UID: "b1729574-6142-4180-a046-a93ccd733932"). InnerVolumeSpecName "kube-api-access-qsnzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:50.790428 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.790392 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1729574-6142-4180-a046-a93ccd733932-tls-cert" pod "b1729574-6142-4180-a046-a93ccd733932" (UID: "b1729574-6142-4180-a046-a93ccd733932"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:50.878307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.878243 5978 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1729574-6142-4180-a046-a93ccd733932-tls-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:36:50.878307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:50.878281 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsnzm\" (UniqueName: \"kubernetes.io/projected/b1729574-6142-4180-a046-a93ccd733932-kube-api-access-qsnzm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:36:51.088366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.088326 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:51.164080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.163600 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vd2tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 00:36:51.164080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.163680 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 00:36:51.243383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.242995 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:51.288804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.288762 5978 generic.go:356] "Generic (PLEG): container finished" podID="b1729574-6142-4180-a046-a93ccd733932" containerID="b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb" exitCode=0 Oct 31 00:36:51.289337 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.288819 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d45fccc5-xwt25" Oct 31 00:36:51.289337 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.288859 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d45fccc5-xwt25" event={"ID":"b1729574-6142-4180-a046-a93ccd733932","Type":"ContainerDied","Data":"b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb"} Oct 31 00:36:51.289337 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.288912 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d45fccc5-xwt25" event={"ID":"b1729574-6142-4180-a046-a93ccd733932","Type":"ContainerDied","Data":"16b733ebefb328947cf2aee90e03d3f31315c5467019ebbf756a86e66cc1921b"} Oct 31 00:36:51.289337 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.288935 5978 scope.go:122] "RemoveContainer" containerID="b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb" Oct 31 00:36:51.303901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.303875 5978 scope.go:122] "RemoveContainer" containerID="b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb" Oct 31 00:36:51.304269 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:36:51.304237 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb\": container with ID starting with b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb not found: ID does not exist" containerID="b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb" Oct 31 00:36:51.304361 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.304282 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb"} err="failed to get container status \"b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb\": rpc error: code = NotFound desc = could not find container \"b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb\": container with ID starting with b2c88ff8fc89ff250ac760eb23a87d2a099cb972e6feb920c737c286269506fb not found: ID does not exist" Oct 31 00:36:51.342539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.336123 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54d45fccc5-xwt25"] Oct 31 00:36:51.348429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:51.348203 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54d45fccc5-xwt25"] Oct 31 00:36:52.162378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:52.162324 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vd2tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 00:36:52.162597 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:52.162404 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 00:36:52.425695 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:52.425601 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1729574-6142-4180-a046-a93ccd733932" path="/var/lib/kubelet/pods/b1729574-6142-4180-a046-a93ccd733932/volumes" Oct 31 00:36:53.162651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:53.162607 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vd2tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 00:36:53.162858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:53.162695 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 00:36:54.168766 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:54.168725 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:54.169816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:54.169792 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:54.173397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:54.173370 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 00:36:55.353691 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.353638 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:55.426406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.426372 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-7vfd8"] Oct 31 00:36:55.426976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.426949 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b1729574-6142-4180-a046-a93ccd733932" containerName="authorino" Oct 31 00:36:55.427182 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.427146 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1729574-6142-4180-a046-a93ccd733932" containerName="authorino" Oct 31 00:36:55.427370 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.427352 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1729574-6142-4180-a046-a93ccd733932" containerName="authorino" Oct 31 00:36:55.433756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.433730 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:36:55.450492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.450447 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 00:36:55.452042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.452017 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-7vfd8"] Oct 31 00:36:55.460313 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.460288 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dtl5x\"" Oct 31 00:36:55.522273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.522239 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:55.533412 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.533377 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33311d84-f2dc-4d32-b52a-9577dba46928-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-7vfd8\" (UID: \"33311d84-f2dc-4d32-b52a-9577dba46928\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:36:55.533597 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.533531 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfvpp\" (UniqueName: \"kubernetes.io/projected/33311d84-f2dc-4d32-b52a-9577dba46928-kube-api-access-sfvpp\") pod \"payload-processing-55fbbbf9dc-7vfd8\" (UID: \"33311d84-f2dc-4d32-b52a-9577dba46928\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:36:55.634566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.634428 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvpp\" (UniqueName: \"kubernetes.io/projected/33311d84-f2dc-4d32-b52a-9577dba46928-kube-api-access-sfvpp\") pod \"payload-processing-55fbbbf9dc-7vfd8\" (UID: \"33311d84-f2dc-4d32-b52a-9577dba46928\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:36:55.634756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.634600 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33311d84-f2dc-4d32-b52a-9577dba46928-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-7vfd8\" (UID: \"33311d84-f2dc-4d32-b52a-9577dba46928\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:36:55.635496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.635472 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33311d84-f2dc-4d32-b52a-9577dba46928-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-7vfd8\" (UID: \"33311d84-f2dc-4d32-b52a-9577dba46928\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:36:55.656608 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.656572 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvpp\" (UniqueName: \"kubernetes.io/projected/33311d84-f2dc-4d32-b52a-9577dba46928-kube-api-access-sfvpp\") pod \"payload-processing-55fbbbf9dc-7vfd8\" (UID: \"33311d84-f2dc-4d32-b52a-9577dba46928\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:36:55.751266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:55.751229 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:36:56.063350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.063318 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-7vfd8"] Oct 31 00:36:56.065127 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:36:56.065091 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33311d84_f2dc_4d32_b52a_9577dba46928.slice/crio-d1b9fa2e3dc3cb0b07a465aba7d67e303c9c47f03ab4e7f25366f7fd019e5daf WatchSource:0}: Error finding container d1b9fa2e3dc3cb0b07a465aba7d67e303c9c47f03ab4e7f25366f7fd019e5daf: Status 404 returned error can't find the container with id d1b9fa2e3dc3cb0b07a465aba7d67e303c9c47f03ab4e7f25366f7fd019e5daf Oct 31 00:36:56.329297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.329259 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" event={"ID":"33311d84-f2dc-4d32-b52a-9577dba46928","Type":"ContainerStarted","Data":"d1b9fa2e3dc3cb0b07a465aba7d67e303c9c47f03ab4e7f25366f7fd019e5daf"} Oct 31 00:36:56.861376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.859738 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h"] Oct 31 00:36:56.865972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.865940 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:56.895715 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.895681 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h"] Oct 31 00:36:56.953084 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.953050 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:56.953274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.953135 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:56.953274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.953169 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:56.953274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.953191 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:56.953274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.953216 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:56.953274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.953248 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60da7d24-5e23-416c-9511-c8b9a7869ee9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:56.953274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.953273 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfnw\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-kube-api-access-hgfnw\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:56.953666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.953301 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:56.953666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:56.953380 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.054828 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.054794 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.055025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.054836 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.055025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.054865 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.055025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.054890 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.055025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.054926 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60da7d24-5e23-416c-9511-c8b9a7869ee9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.055025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.054957 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfnw\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-kube-api-access-hgfnw\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.055025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.054986 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.055323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.055060 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.055323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.055107 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.055598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.055575 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.056932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.056220 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.056932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.056619 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60da7d24-5e23-416c-9511-c8b9a7869ee9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.056932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.056892 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.060681 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.060650 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.062985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.062943 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.081007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.080929 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.081874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.081820 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfnw\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-kube-api-access-hgfnw\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.082623 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.082597 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-97z9h\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.189015 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.188928 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:57.523696 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.523548 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h"] Oct 31 00:36:57.532923 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:36:57.532883 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60da7d24_5e23_416c_9511_c8b9a7869ee9.slice/crio-2cecc69b5dfed1e19e0cf0c70b0bb19d49d8ec366581f849f4ec1d604fba4166 WatchSource:0}: Error finding container 2cecc69b5dfed1e19e0cf0c70b0bb19d49d8ec366581f849f4ec1d604fba4166: Status 404 returned error can't find the container with id 2cecc69b5dfed1e19e0cf0c70b0bb19d49d8ec366581f849f4ec1d604fba4166 Oct 31 00:36:57.537506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.537436 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:36:57.537639 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.537561 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:36:57.537639 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:57.537614 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:36:58.385416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:58.384552 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" event={"ID":"60da7d24-5e23-416c-9511-c8b9a7869ee9","Type":"ContainerStarted","Data":"85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9"} Oct 31 00:36:58.385416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:58.384617 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" event={"ID":"60da7d24-5e23-416c-9511-c8b9a7869ee9","Type":"ContainerStarted","Data":"2cecc69b5dfed1e19e0cf0c70b0bb19d49d8ec366581f849f4ec1d604fba4166"} Oct 31 00:36:59.189204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:59.189173 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:36:59.190990 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:59.190957 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 31 00:36:59.191142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:59.191026 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 31 00:36:59.392136 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:59.392038 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" event={"ID":"33311d84-f2dc-4d32-b52a-9577dba46928","Type":"ContainerStarted","Data":"75522c9f1fe133b39ecaff1cd28848a11b5770b3a2589d114de3c25f545b9aec"} Oct 31 00:36:59.392570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:59.392240 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:36:59.437241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:59.437178 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podStartSLOduration=3.437161528 podStartE2EDuration="3.437161528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:58.447705342 +0000 UTC m=+1036.145819108" watchObservedRunningTime="2026-10-31 00:36:59.437161528 +0000 UTC m=+1037.135275293" Oct 31 00:36:59.437444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:36:59.437313 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" podStartSLOduration=1.427398537 podStartE2EDuration="4.437306764s" totalImagesPullingTime="3.009908227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:59.435870553 +0000 UTC m=+1037.133984329" watchObservedRunningTime="2026-10-31 00:36:59.437306764 +0000 UTC m=+1037.135420534" Oct 31 00:37:00.189823 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:00.189782 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 31 00:37:00.190007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:00.189842 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 31 00:37:01.189732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:01.189689 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 31 00:37:01.190137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:01.189754 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 31 00:37:02.189839 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:02.189799 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 31 00:37:02.190233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:02.189855 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 31 00:37:03.180245 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:37:03.180205 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:37:03.189222 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:03.189196 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 31 00:37:03.189305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:03.189237 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 31 00:37:04.190159 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:04.190123 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 31 00:37:04.190608 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:04.190180 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 31 00:37:05.190219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:05.190184 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 31 00:37:05.190673 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:05.190254 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 31 00:37:06.193662 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:06.193628 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:37:06.194144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:06.193905 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:37:06.194806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:06.194783 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 00:37:10.400131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:10.400099 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-7vfd8" Oct 31 00:37:11.866086 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.866050 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr"] Oct 31 00:37:11.869064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.869037 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:11.897407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.897372 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr"] Oct 31 00:37:11.965019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.964978 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:11.965019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.965021 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:11.965277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.965121 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m2gj\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-kube-api-access-4m2gj\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:11.965277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.965178 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:11.965363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.965281 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:11.965363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.965303 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:11.965363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.965335 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:11.965490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.965404 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:11.965490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:11.965427 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066253 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066307 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066334 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066379 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m2gj\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-kube-api-access-4m2gj\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066426 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066563 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066583 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066641 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066678 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.066918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066849 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.067016 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.066988 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.067152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.067019 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.067217 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.067194 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.067419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.067398 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.068637 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.068612 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.068860 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.068841 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.087784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.087735 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.087945 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.087808 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m2gj\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-kube-api-access-4m2gj\") pod \"maas-default-gateway-openshift-default-6886b96f9-62hfr\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.182515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.182429 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:12.501060 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.501031 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr"] Oct 31 00:37:12.503447 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:12.503408 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6686fe8f_e8e9_4f29_8760_efd0d88ac1cc.slice/crio-9db98b73a3ba2771cd763397bc177423851d84e22e81283bef1ea39a4e6881a2 WatchSource:0}: Error finding container 9db98b73a3ba2771cd763397bc177423851d84e22e81283bef1ea39a4e6881a2: Status 404 returned error can't find the container with id 9db98b73a3ba2771cd763397bc177423851d84e22e81283bef1ea39a4e6881a2 Oct 31 00:37:12.506624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.506588 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:12.506705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.506660 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:12.506705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:12.506690 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:13.459798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:13.459759 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" event={"ID":"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc","Type":"ContainerStarted","Data":"f9683bb7900f277e80a81a2293d27e40c17718831f329dc409440c514b7510c4"} Oct 31 00:37:13.460197 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:13.459826 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" event={"ID":"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc","Type":"ContainerStarted","Data":"9db98b73a3ba2771cd763397bc177423851d84e22e81283bef1ea39a4e6881a2"} Oct 31 00:37:14.183715 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:14.183670 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:14.185176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:14.185150 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:37:14.185300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:14.185205 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:37:15.183518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:15.183484 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:37:15.183922 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:15.183541 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:37:16.183141 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:16.183098 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:37:16.183330 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:16.183161 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:37:17.183581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:17.183541 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:37:17.184006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:17.183601 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:37:18.183408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:18.183368 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:37:18.183611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:18.183432 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:37:19.183690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:19.183647 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:37:19.184087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:19.183725 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:37:19.990250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:19.990203 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:20.501807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:20.501767 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:20.878265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:20.878226 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:20.878618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:20.878595 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:20.879395 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:20.879375 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 00:37:20.938571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:20.938502 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" podStartSLOduration=9.938484621 podStartE2EDuration="9.938484621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:13.529797829 +0000 UTC m=+1051.227911595" watchObservedRunningTime="2026-10-31 00:37:20.938484621 +0000 UTC m=+1058.636598388" Oct 31 00:37:22.184369 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:22.184317 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:22.340556 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:22.340519 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:23.740244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:23.740203 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:23.908517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:23.908480 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:33.181026 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:37:33.180990 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:37:43.542809 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:43.542738 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:46.159858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:46.159825 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:46.321298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:46.321253 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:46.656216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:46.656176 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:46.804896 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:46.804851 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:46.953982 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:46.953889 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.128830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.128792 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.171563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.171489 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd"] Oct 31 00:37:47.179677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.179640 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.205709 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.205623 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd"] Oct 31 00:37:47.228684 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.228651 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-671959-openshift-default-dockercfg-mgz7q\"" Oct 31 00:37:47.272802 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.272761 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.272997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.272821 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.272997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.272952 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.273071 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.272994 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.273138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.273121 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.273185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.273168 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98cct\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-kube-api-access-98cct\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.273225 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.273201 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.273258 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.273236 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.273326 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.273309 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.325629 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.325582 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.342774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.342741 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5"] Oct 31 00:37:47.346881 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.346855 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.363405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.363374 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-be301b-openshift-default-dockercfg-4bp8d\"" Oct 31 00:37:47.374277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.374235 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.374540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.374286 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98cct\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-kube-api-access-98cct\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.374540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.374327 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.374540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.374376 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.374540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.374407 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.374540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.374480 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.374540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.374527 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.375188 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.374630 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.375188 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.374677 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.375276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.375253 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5"] Oct 31 00:37:47.375444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.375377 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.375949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.375920 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.375949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.375935 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.376104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.375995 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.376104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.376043 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.378076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.378053 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.378301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.378282 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.408109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.408068 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.408281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.408256 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98cct\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-kube-api-access-98cct\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.476241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.476150 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.476241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.476187 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.476241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.476214 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.476543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.476364 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.476543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.476409 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.476543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.476497 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.476543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.476535 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzhn\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-kube-api-access-7tzhn\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.476741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.476566 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.476741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.476673 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.495628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.495593 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:47.500488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.500224 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.501774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.501694 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw"] Oct 31 00:37:47.506413 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.506176 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.518994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.518962 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0d8bae-openshift-default-dockercfg-bvb9r\"" Oct 31 00:37:47.534896 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.534842 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw"] Oct 31 00:37:47.577558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577518 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.577725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577574 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.577725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577612 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ed5b38-29a4-42e8-933f-c7b305641872-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.577725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577640 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwkj\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-kube-api-access-7pwkj\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.577725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577678 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.577725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577715 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzhn\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-kube-api-access-7tzhn\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.578000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577762 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.578000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577812 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.578000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577841 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.578000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577890 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.578000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577958 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.578000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.577989 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.578290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.578005 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.578290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.578014 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.578290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.578045 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.578290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.578087 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.578290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.578132 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.578290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.578159 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.578290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.578194 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.578754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.578729 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.579602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.579446 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.579602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.579567 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.580244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.580218 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.581126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.581097 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.581285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.581132 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.600095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.600065 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.600343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.600321 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzhn\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-kube-api-access-7tzhn\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.662248 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.662205 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:47.665515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.665483 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.678876 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.678844 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.678886 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.678930 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.678975 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679006 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ed5b38-29a4-42e8-933f-c7b305641872-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679031 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwkj\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-kube-api-access-7pwkj\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679065 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679131 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679158 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679322 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679757 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.679910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679757 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.680181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679928 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.680181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.679992 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ed5b38-29a4-42e8-933f-c7b305641872-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.682323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.682302 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.682434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.682340 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.700528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.700487 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.700682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.700493 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwkj\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-kube-api-access-7pwkj\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.822187 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.822152 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd"] Oct 31 00:37:47.823726 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:47.823700 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fa924f_1429_48ed_8ca2_bb5a3fca1f92.slice/crio-366ceebf5dad1eda39ada126d9e084ec176ff3d7de43ba28f5e362a8ef9ad29f WatchSource:0}: Error finding container 366ceebf5dad1eda39ada126d9e084ec176ff3d7de43ba28f5e362a8ef9ad29f: Status 404 returned error can't find the container with id 366ceebf5dad1eda39ada126d9e084ec176ff3d7de43ba28f5e362a8ef9ad29f Oct 31 00:37:47.826039 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.826004 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.826148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.826127 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:47.826206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.826192 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:47.826238 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.826230 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:47.839008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.838988 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:47.968967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.968847 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5"] Oct 31 00:37:47.972368 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:47.972337 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a07c21_7ad3_4153_a7fe_a5ea3e263cc6.slice/crio-59c3b48ff29cc1db712faf1475ece7f83747f0853ff937ce822dab6d673177ee WatchSource:0}: Error finding container 59c3b48ff29cc1db712faf1475ece7f83747f0853ff937ce822dab6d673177ee: Status 404 returned error can't find the container with id 59c3b48ff29cc1db712faf1475ece7f83747f0853ff937ce822dab6d673177ee Oct 31 00:37:47.975527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.975493 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:47.975613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.975560 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:47.975613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:47.975595 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:48.144173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.144139 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw"] Oct 31 00:37:48.148326 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:48.148296 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ed5b38_29a4_42e8_933f_c7b305641872.slice/crio-0919e2872bb2c696968f180441f9bdcba3d53f1acc82a2dd5bf4cd7378c2f5ff WatchSource:0}: Error finding container 0919e2872bb2c696968f180441f9bdcba3d53f1acc82a2dd5bf4cd7378c2f5ff: Status 404 returned error can't find the container with id 0919e2872bb2c696968f180441f9bdcba3d53f1acc82a2dd5bf4cd7378c2f5ff Oct 31 00:37:48.150779 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.150741 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:48.150887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.150818 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:48.150887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.150847 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:48.646239 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.646198 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" event={"ID":"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6","Type":"ContainerStarted","Data":"df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb"} Oct 31 00:37:48.646700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.646250 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" event={"ID":"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6","Type":"ContainerStarted","Data":"59c3b48ff29cc1db712faf1475ece7f83747f0853ff937ce822dab6d673177ee"} Oct 31 00:37:48.647817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.647790 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" event={"ID":"a7ed5b38-29a4-42e8-933f-c7b305641872","Type":"ContainerStarted","Data":"750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4"} Oct 31 00:37:48.647944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.647839 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" event={"ID":"a7ed5b38-29a4-42e8-933f-c7b305641872","Type":"ContainerStarted","Data":"0919e2872bb2c696968f180441f9bdcba3d53f1acc82a2dd5bf4cd7378c2f5ff"} Oct 31 00:37:48.649317 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.649289 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" event={"ID":"65fa924f-1429-48ed-8ca2-bb5a3fca1f92","Type":"ContainerStarted","Data":"f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d"} Oct 31 00:37:48.649447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.649337 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" event={"ID":"65fa924f-1429-48ed-8ca2-bb5a3fca1f92","Type":"ContainerStarted","Data":"366ceebf5dad1eda39ada126d9e084ec176ff3d7de43ba28f5e362a8ef9ad29f"} Oct 31 00:37:48.662629 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.662603 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:48.687259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.687202 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" podStartSLOduration=1.687187816 podStartE2EDuration="1.687187816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:48.686209055 +0000 UTC m=+1086.384322820" watchObservedRunningTime="2026-10-31 00:37:48.687187816 +0000 UTC m=+1086.385301580" Oct 31 00:37:48.731278 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.731213 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" podStartSLOduration=1.731197181 podStartE2EDuration="1.731197181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:48.729202337 +0000 UTC m=+1086.427316126" watchObservedRunningTime="2026-10-31 00:37:48.731197181 +0000 UTC m=+1086.429311016" Oct 31 00:37:48.771711 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.771655 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" podStartSLOduration=1.771638458 podStartE2EDuration="1.771638458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:48.768995329 +0000 UTC m=+1086.467109106" watchObservedRunningTime="2026-10-31 00:37:48.771638458 +0000 UTC m=+1086.469752224" Oct 31 00:37:48.839949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:48.839906 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:49.496849 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:49.496804 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:49.502239 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:49.502210 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:49.655425 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:49.655394 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:49.656420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:49.656399 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:37:49.668061 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:49.668032 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:49.845979 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:49.845936 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:50.085262 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:50.085220 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:50.244717 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:50.244628 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:50.661163 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:50.661125 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:50.661734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:50.661181 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:50.661803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:50.661777 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:37:50.661927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:50.661909 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:37:51.668734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:51.668696 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:51.853805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:51.853767 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:51.929367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:51.929274 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl"] Oct 31 00:37:51.933359 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:51.933334 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:37:51.951147 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:51.951115 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b024b5\"" Oct 31 00:37:51.959905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:51.959874 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b024b5-dockercfg-2gqwt\"" Oct 31 00:37:51.961910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:51.961882 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl"] Oct 31 00:37:51.994377 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:51.994340 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:52.033200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.028929 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj"] Oct 31 00:37:52.033200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.033084 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd77p\" (UniqueName: \"kubernetes.io/projected/f22077e0-eac8-45ef-92fd-ac6f597563fa-kube-api-access-jd77p\") pod \"payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl\" (UID: \"f22077e0-eac8-45ef-92fd-ac6f597563fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:37:52.033594 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.033335 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22077e0-eac8-45ef-92fd-ac6f597563fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl\" (UID: \"f22077e0-eac8-45ef-92fd-ac6f597563fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:37:52.033594 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.033576 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:37:52.055061 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.055021 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj"] Oct 31 00:37:52.134869 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.134835 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd77p\" (UniqueName: \"kubernetes.io/projected/f22077e0-eac8-45ef-92fd-ac6f597563fa-kube-api-access-jd77p\") pod \"payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl\" (UID: \"f22077e0-eac8-45ef-92fd-ac6f597563fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:37:52.135046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.134903 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj\" (UID: \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:37:52.135046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.134927 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8qsl\" (UniqueName: \"kubernetes.io/projected/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-kube-api-access-q8qsl\") pod \"payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj\" (UID: \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:37:52.135242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.135068 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22077e0-eac8-45ef-92fd-ac6f597563fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl\" (UID: \"f22077e0-eac8-45ef-92fd-ac6f597563fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:37:52.135773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.135753 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22077e0-eac8-45ef-92fd-ac6f597563fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl\" (UID: \"f22077e0-eac8-45ef-92fd-ac6f597563fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:37:52.164323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.164288 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd77p\" (UniqueName: \"kubernetes.io/projected/f22077e0-eac8-45ef-92fd-ac6f597563fa-kube-api-access-jd77p\") pod \"payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl\" (UID: \"f22077e0-eac8-45ef-92fd-ac6f597563fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:37:52.165838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.165812 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:52.236017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.235927 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj\" (UID: \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:37:52.236017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.235968 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8qsl\" (UniqueName: \"kubernetes.io/projected/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-kube-api-access-q8qsl\") pod \"payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj\" (UID: \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:37:52.236750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.236725 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj\" (UID: \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:37:52.247338 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.247310 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:37:52.257644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.257612 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8qsl\" (UniqueName: \"kubernetes.io/projected/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-kube-api-access-q8qsl\") pod \"payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj\" (UID: \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:37:52.310627 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.310186 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:52.351921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.351887 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:37:52.496553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.496432 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:52.592299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.592273 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl"] Oct 31 00:37:52.593742 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:52.593715 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22077e0_eac8_45ef_92fd_ac6f597563fa.slice/crio-e3ae887ff2a391d90735f99dc543b8d7f4b2fe1c9f83e4b397f1f80c7b0ee215 WatchSource:0}: Error finding container e3ae887ff2a391d90735f99dc543b8d7f4b2fe1c9f83e4b397f1f80c7b0ee215: Status 404 returned error can't find the container with id e3ae887ff2a391d90735f99dc543b8d7f4b2fe1c9f83e4b397f1f80c7b0ee215 Oct 31 00:37:52.664150 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.664125 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj"] Oct 31 00:37:52.666203 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:52.666177 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95d49d2_1d4c_49cf_9409_2b6e3bd7c6ee.slice/crio-1f0b9057c32128721460588ed8e23dc835d591a6d43753147aa91e43f8bc2229 WatchSource:0}: Error finding container 1f0b9057c32128721460588ed8e23dc835d591a6d43753147aa91e43f8bc2229: Status 404 returned error can't find the container with id 1f0b9057c32128721460588ed8e23dc835d591a6d43753147aa91e43f8bc2229 Oct 31 00:37:52.673933 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.673899 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" event={"ID":"f22077e0-eac8-45ef-92fd-ac6f597563fa","Type":"ContainerStarted","Data":"6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6"} Oct 31 00:37:52.674829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.673967 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" event={"ID":"f22077e0-eac8-45ef-92fd-ac6f597563fa","Type":"ContainerStarted","Data":"e3ae887ff2a391d90735f99dc543b8d7f4b2fe1c9f83e4b397f1f80c7b0ee215"} Oct 31 00:37:52.674829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.674482 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:37:52.710332 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.710299 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:52.723382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.718403 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" podStartSLOduration=1.718387125 podStartE2EDuration="1.718387125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:52.717293315 +0000 UTC m=+1090.415407079" watchObservedRunningTime="2026-10-31 00:37:52.718387125 +0000 UTC m=+1090.416500890" Oct 31 00:37:52.891139 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:52.891106 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.053050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.053006 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.211020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.210923 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.680952 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.680902 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" event={"ID":"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee","Type":"ContainerStarted","Data":"609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df"} Oct 31 00:37:53.681402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.680973 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" event={"ID":"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee","Type":"ContainerStarted","Data":"1f0b9057c32128721460588ed8e23dc835d591a6d43753147aa91e43f8bc2229"} Oct 31 00:37:53.681402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.681130 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:37:53.728164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.728095 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" podStartSLOduration=1.728073256 podStartE2EDuration="1.728073256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:53.725135324 +0000 UTC m=+1091.423249079" watchObservedRunningTime="2026-10-31 00:37:53.728073256 +0000 UTC m=+1091.426187023" Oct 31 00:37:53.864013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.863788 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n"] Oct 31 00:37:53.868208 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.868177 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:53.891047 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.890978 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cckh7\"" Oct 31 00:37:53.891260 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.891070 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:37:53.891260 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.891181 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:37:53.891260 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.891212 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:37:53.891400 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.891004 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-daaf26-serving-cert\"" Oct 31 00:37:53.898865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.898832 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n"] Oct 31 00:37:53.915506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.915448 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.955975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.955887 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f07dc-e69f-4e51-b1de-178d98232462-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:53.955975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.955938 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-metrics-certs\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:53.956174 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.956047 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzkj\" (UniqueName: \"kubernetes.io/projected/440f07dc-e69f-4e51-b1de-178d98232462-kube-api-access-zmzkj\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:53.956174 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:53.956109 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.056775 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.056737 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.056974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.056866 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f07dc-e69f-4e51-b1de-178d98232462-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.056974 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:37:54.056876 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-serving-cert: secret "maas-api-e2e-worker-w3-daaf26-serving-cert" not found Oct 31 00:37:54.056974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.056902 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-metrics-certs\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.056974 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:37:54.056952 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls podName:440f07dc-e69f-4e51-b1de-178d98232462 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:54.556934727 +0000 UTC m=+1092.255048470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls") pod "maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" (UID: "440f07dc-e69f-4e51-b1de-178d98232462") : secret "maas-api-e2e-worker-w3-daaf26-serving-cert" not found Oct 31 00:37:54.057216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.056985 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzkj\" (UniqueName: \"kubernetes.io/projected/440f07dc-e69f-4e51-b1de-178d98232462-kube-api-access-zmzkj\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.057517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.057493 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f07dc-e69f-4e51-b1de-178d98232462-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.059497 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.059472 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-metrics-certs\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.062778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.062754 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f"] Oct 31 00:37:54.066605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.066586 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:37:54.081068 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.081043 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-daaf26\"" Oct 31 00:37:54.081209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.081137 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-daaf26-dockercfg-zgb6r\"" Oct 31 00:37:54.083342 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.083322 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzkj\" (UniqueName: \"kubernetes.io/projected/440f07dc-e69f-4e51-b1de-178d98232462-kube-api-access-zmzkj\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.085142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.085116 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:54.087774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.087746 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f"] Oct 31 00:37:54.135256 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.135217 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm"] Oct 31 00:37:54.139327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.139301 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:37:54.158233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.158188 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2mv\" (UniqueName: \"kubernetes.io/projected/47fce07c-1185-466e-be3b-7598826ddccc-kube-api-access-vx2mv\") pod \"payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f\" (UID: \"47fce07c-1185-466e-be3b-7598826ddccc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:37:54.158447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.158424 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fce07c-1185-466e-be3b-7598826ddccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f\" (UID: \"47fce07c-1185-466e-be3b-7598826ddccc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:37:54.160674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.160646 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm"] Oct 31 00:37:54.244527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.243767 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:54.259659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.259627 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758d1b69-ab65-4337-9e7d-d24e7b5feda6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm\" (UID: \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:37:54.259659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.259662 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8fk\" (UniqueName: \"kubernetes.io/projected/758d1b69-ab65-4337-9e7d-d24e7b5feda6-kube-api-access-2m8fk\") pod \"payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm\" (UID: \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:37:54.259873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.259751 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fce07c-1185-466e-be3b-7598826ddccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f\" (UID: \"47fce07c-1185-466e-be3b-7598826ddccc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:37:54.259873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.259836 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2mv\" (UniqueName: \"kubernetes.io/projected/47fce07c-1185-466e-be3b-7598826ddccc-kube-api-access-vx2mv\") pod \"payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f\" (UID: \"47fce07c-1185-466e-be3b-7598826ddccc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:37:54.260660 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.260639 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fce07c-1185-466e-be3b-7598826ddccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f\" (UID: \"47fce07c-1185-466e-be3b-7598826ddccc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:37:54.281218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.281186 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2mv\" (UniqueName: \"kubernetes.io/projected/47fce07c-1185-466e-be3b-7598826ddccc-kube-api-access-vx2mv\") pod \"payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f\" (UID: \"47fce07c-1185-466e-be3b-7598826ddccc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:37:54.361372 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.361329 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758d1b69-ab65-4337-9e7d-d24e7b5feda6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm\" (UID: \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:37:54.361372 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.361372 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8fk\" (UniqueName: \"kubernetes.io/projected/758d1b69-ab65-4337-9e7d-d24e7b5feda6-kube-api-access-2m8fk\") pod \"payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm\" (UID: \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:37:54.362101 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.362076 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758d1b69-ab65-4337-9e7d-d24e7b5feda6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm\" (UID: \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:37:54.381322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.381288 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:37:54.383152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.383128 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8fk\" (UniqueName: \"kubernetes.io/projected/758d1b69-ab65-4337-9e7d-d24e7b5feda6-kube-api-access-2m8fk\") pod \"payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm\" (UID: \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:37:54.454092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.453548 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:37:54.563520 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.563474 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.566176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.566139 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls\") pod \"maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:54.681350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.681161 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f"] Oct 31 00:37:54.683062 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:54.683026 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fce07c_1185_466e_be3b_7598826ddccc.slice/crio-a80164dd239928c4e1b380c4dd1b181cff107b3dd8c0be3e92bd398ef0c4f50f WatchSource:0}: Error finding container a80164dd239928c4e1b380c4dd1b181cff107b3dd8c0be3e92bd398ef0c4f50f: Status 404 returned error can't find the container with id a80164dd239928c4e1b380c4dd1b181cff107b3dd8c0be3e92bd398ef0c4f50f Oct 31 00:37:54.758785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.758759 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm"] Oct 31 00:37:54.760525 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:54.760499 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758d1b69_ab65_4337_9e7d_d24e7b5feda6.slice/crio-457ecf4528af6d6012888b674e3881792d59d9aa8b1940a9173c52a3a59f9e31 WatchSource:0}: Error finding container 457ecf4528af6d6012888b674e3881792d59d9aa8b1940a9173c52a3a59f9e31: Status 404 returned error can't find the container with id 457ecf4528af6d6012888b674e3881792d59d9aa8b1940a9173c52a3a59f9e31 Oct 31 00:37:54.782402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:54.782378 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:55.099823 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.099788 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n"] Oct 31 00:37:55.101834 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:55.101809 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440f07dc_e69f_4e51_b1de_178d98232462.slice/crio-4e12b0ee43dc6c49f1cf014bb128c3591e4e698004f2ff847860dbd1c0fac880 WatchSource:0}: Error finding container 4e12b0ee43dc6c49f1cf014bb128c3591e4e698004f2ff847860dbd1c0fac880: Status 404 returned error can't find the container with id 4e12b0ee43dc6c49f1cf014bb128c3591e4e698004f2ff847860dbd1c0fac880 Oct 31 00:37:55.696036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.695999 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" event={"ID":"440f07dc-e69f-4e51-b1de-178d98232462","Type":"ContainerStarted","Data":"4e12b0ee43dc6c49f1cf014bb128c3591e4e698004f2ff847860dbd1c0fac880"} Oct 31 00:37:55.698606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.698576 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" event={"ID":"758d1b69-ab65-4337-9e7d-d24e7b5feda6","Type":"ContainerStarted","Data":"8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316"} Oct 31 00:37:55.698780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.698634 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" event={"ID":"758d1b69-ab65-4337-9e7d-d24e7b5feda6","Type":"ContainerStarted","Data":"457ecf4528af6d6012888b674e3881792d59d9aa8b1940a9173c52a3a59f9e31"} Oct 31 00:37:55.698780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.698674 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:37:55.701230 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.701202 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" event={"ID":"47fce07c-1185-466e-be3b-7598826ddccc","Type":"ContainerStarted","Data":"8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710"} Oct 31 00:37:55.701350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.701252 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" event={"ID":"47fce07c-1185-466e-be3b-7598826ddccc","Type":"ContainerStarted","Data":"a80164dd239928c4e1b380c4dd1b181cff107b3dd8c0be3e92bd398ef0c4f50f"} Oct 31 00:37:55.701350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.701290 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:37:55.741176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.741110 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" podStartSLOduration=1.741088733 podStartE2EDuration="1.741088733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:55.738206682 +0000 UTC m=+1093.436320447" watchObservedRunningTime="2026-10-31 00:37:55.741088733 +0000 UTC m=+1093.439202500" Oct 31 00:37:55.774946 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.774859 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" podStartSLOduration=1.774842746 podStartE2EDuration="1.774842746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:55.773813758 +0000 UTC m=+1093.471927523" watchObservedRunningTime="2026-10-31 00:37:55.774842746 +0000 UTC m=+1093.472956510" Oct 31 00:37:55.954111 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:55.954014 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:56.123474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.122443 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:56.539906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.539832 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:56.669904 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.669863 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5"] Oct 31 00:37:56.678924 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.678893 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:37:56.691956 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.691927 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-735123\"" Oct 31 00:37:56.692273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.692256 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-735123-dockercfg-bt5z4\"" Oct 31 00:37:56.694750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.694719 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5"] Oct 31 00:37:56.696310 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.696272 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:56.785242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.785037 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w"] Oct 31 00:37:56.794062 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.793974 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2nbs\" (UniqueName: \"kubernetes.io/projected/e21d5451-a5c3-4f7e-8fe0-fae74204a205-kube-api-access-r2nbs\") pod \"payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5\" (UID: \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:37:56.794308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.794284 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21d5451-a5c3-4f7e-8fe0-fae74204a205-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5\" (UID: \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:37:56.797285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.797255 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:37:56.811293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.810070 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w"] Oct 31 00:37:56.895758 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.895713 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwh2\" (UniqueName: \"kubernetes.io/projected/415e9e8c-2289-468b-9d49-33498212dd0f-kube-api-access-2xwh2\") pod \"payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w\" (UID: \"415e9e8c-2289-468b-9d49-33498212dd0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:37:56.895940 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.895814 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21d5451-a5c3-4f7e-8fe0-fae74204a205-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5\" (UID: \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:37:56.896018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.895986 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2nbs\" (UniqueName: \"kubernetes.io/projected/e21d5451-a5c3-4f7e-8fe0-fae74204a205-kube-api-access-r2nbs\") pod \"payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5\" (UID: \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:37:56.896106 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.896085 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e9e8c-2289-468b-9d49-33498212dd0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w\" (UID: \"415e9e8c-2289-468b-9d49-33498212dd0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:37:56.896700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.896679 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21d5451-a5c3-4f7e-8fe0-fae74204a205-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5\" (UID: \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:37:56.918248 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.918213 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2nbs\" (UniqueName: \"kubernetes.io/projected/e21d5451-a5c3-4f7e-8fe0-fae74204a205-kube-api-access-r2nbs\") pod \"payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5\" (UID: \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:37:56.997555 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.997517 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e9e8c-2289-468b-9d49-33498212dd0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w\" (UID: \"415e9e8c-2289-468b-9d49-33498212dd0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:37:56.997762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.997594 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwh2\" (UniqueName: \"kubernetes.io/projected/415e9e8c-2289-468b-9d49-33498212dd0f-kube-api-access-2xwh2\") pod \"payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w\" (UID: \"415e9e8c-2289-468b-9d49-33498212dd0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:37:56.997762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.997718 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:37:56.998308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:56.998287 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e9e8c-2289-468b-9d49-33498212dd0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w\" (UID: \"415e9e8c-2289-468b-9d49-33498212dd0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:37:57.019760 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.019686 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwh2\" (UniqueName: \"kubernetes.io/projected/415e9e8c-2289-468b-9d49-33498212dd0f-kube-api-access-2xwh2\") pod \"payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w\" (UID: \"415e9e8c-2289-468b-9d49-33498212dd0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:37:57.117470 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.117415 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:37:57.309733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.309703 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5"] Oct 31 00:37:57.312026 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:57.311994 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21d5451_a5c3_4f7e_8fe0_fae74204a205.slice/crio-d5553919d6be470bf098c79576a1cfd1cc1107543acad91c99a36a8efa166d7c WatchSource:0}: Error finding container d5553919d6be470bf098c79576a1cfd1cc1107543acad91c99a36a8efa166d7c: Status 404 returned error can't find the container with id d5553919d6be470bf098c79576a1cfd1cc1107543acad91c99a36a8efa166d7c Oct 31 00:37:57.430425 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.429950 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w"] Oct 31 00:37:57.433341 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:57.433312 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415e9e8c_2289_468b_9d49_33498212dd0f.slice/crio-5fbd686fbe29c329a8014e1dc02acecd81fb70b39115e653a3c4e15faf6a1e03 WatchSource:0}: Error finding container 5fbd686fbe29c329a8014e1dc02acecd81fb70b39115e653a3c4e15faf6a1e03: Status 404 returned error can't find the container with id 5fbd686fbe29c329a8014e1dc02acecd81fb70b39115e653a3c4e15faf6a1e03 Oct 31 00:37:57.443312 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.443278 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:57.570142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.570101 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w"] Oct 31 00:37:57.574678 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.574648 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:37:57.589228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.589199 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-be301b-dockercfg-tcmkx\"" Oct 31 00:37:57.591770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.591736 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-be301b\"" Oct 31 00:37:57.601151 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.601106 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:57.601375 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.601287 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w"] Oct 31 00:37:57.690234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.689280 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9"] Oct 31 00:37:57.694650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.694623 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:37:57.707830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.707784 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdprp\" (UniqueName: \"kubernetes.io/projected/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-kube-api-access-pdprp\") pod \"payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w\" (UID: \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:37:57.708352 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.707852 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w\" (UID: \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:37:57.726828 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.726794 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9"] Oct 31 00:37:57.733708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.733667 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" event={"ID":"415e9e8c-2289-468b-9d49-33498212dd0f","Type":"ContainerStarted","Data":"80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2"} Oct 31 00:37:57.733906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.733737 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" event={"ID":"415e9e8c-2289-468b-9d49-33498212dd0f","Type":"ContainerStarted","Data":"5fbd686fbe29c329a8014e1dc02acecd81fb70b39115e653a3c4e15faf6a1e03"} Oct 31 00:37:57.734589 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.734569 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:37:57.737416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.737381 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" event={"ID":"e21d5451-a5c3-4f7e-8fe0-fae74204a205","Type":"ContainerStarted","Data":"b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240"} Oct 31 00:37:57.737568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.737444 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" event={"ID":"e21d5451-a5c3-4f7e-8fe0-fae74204a205","Type":"ContainerStarted","Data":"d5553919d6be470bf098c79576a1cfd1cc1107543acad91c99a36a8efa166d7c"} Oct 31 00:37:57.737773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.737753 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:37:57.770855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.770803 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" podStartSLOduration=1.7707852659999999 podStartE2EDuration="1.770785266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:57.769858486 +0000 UTC m=+1095.467972252" watchObservedRunningTime="2026-10-31 00:37:57.770785266 +0000 UTC m=+1095.468899031" Oct 31 00:37:57.808558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.808487 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" podStartSLOduration=1.8084661560000002 podStartE2EDuration="1.808466156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:57.8056746 +0000 UTC m=+1095.503788365" watchObservedRunningTime="2026-10-31 00:37:57.808466156 +0000 UTC m=+1095.506579916" Oct 31 00:37:57.809284 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.809255 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc746f8f-fe26-4bd8-99da-cee03ed4f520-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9\" (UID: \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:37:57.809407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.809314 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdprp\" (UniqueName: \"kubernetes.io/projected/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-kube-api-access-pdprp\") pod \"payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w\" (UID: \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:37:57.809407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.809389 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w\" (UID: \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:37:57.809567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.809524 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbzn\" (UniqueName: \"kubernetes.io/projected/fc746f8f-fe26-4bd8-99da-cee03ed4f520-kube-api-access-tbbzn\") pod \"payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9\" (UID: \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:37:57.810186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.810161 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w\" (UID: \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:37:57.830937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.830905 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdprp\" (UniqueName: \"kubernetes.io/projected/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-kube-api-access-pdprp\") pod \"payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w\" (UID: \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:37:57.890924 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.890833 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:37:57.910432 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.910399 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc746f8f-fe26-4bd8-99da-cee03ed4f520-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9\" (UID: \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:37:57.910624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.910603 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbzn\" (UniqueName: \"kubernetes.io/projected/fc746f8f-fe26-4bd8-99da-cee03ed4f520-kube-api-access-tbbzn\") pod \"payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9\" (UID: \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:37:57.911356 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.911332 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc746f8f-fe26-4bd8-99da-cee03ed4f520-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9\" (UID: \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:37:57.932612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:57.932577 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbzn\" (UniqueName: \"kubernetes.io/projected/fc746f8f-fe26-4bd8-99da-cee03ed4f520-kube-api-access-tbbzn\") pod \"payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9\" (UID: \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:37:58.011871 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.011820 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:37:58.138829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.138790 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:58.200212 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.200179 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w"] Oct 31 00:37:58.202063 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:58.202026 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ef4823_0af5_4fa5_a498_9eab4a8fc4c4.slice/crio-ef587facbc1e9fe9625aa0c1c6a5cce3b01c0d0b0bc120b743dd7810377cbe43 WatchSource:0}: Error finding container ef587facbc1e9fe9625aa0c1c6a5cce3b01c0d0b0bc120b743dd7810377cbe43: Status 404 returned error can't find the container with id ef587facbc1e9fe9625aa0c1c6a5cce3b01c0d0b0bc120b743dd7810377cbe43 Oct 31 00:37:58.316183 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.316075 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:58.325739 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:58.325694 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc746f8f_fe26_4bd8_99da_cee03ed4f520.slice/crio-87ecbed0fbeddf1171a8ec3a10913d51d0b45cd601063ab396fb608707b9c62c WatchSource:0}: Error finding container 87ecbed0fbeddf1171a8ec3a10913d51d0b45cd601063ab396fb608707b9c62c: Status 404 returned error can't find the container with id 87ecbed0fbeddf1171a8ec3a10913d51d0b45cd601063ab396fb608707b9c62c Oct 31 00:37:58.328158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.328129 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9"] Oct 31 00:37:58.467922 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.467828 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:58.568896 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.568858 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph"] Oct 31 00:37:58.572877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.572856 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:37:58.586983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.586951 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-c9e377\"" Oct 31 00:37:58.587525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.587496 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-c9e377-dockercfg-n7h9c\"" Oct 31 00:37:58.594600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.594566 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph"] Oct 31 00:37:58.615914 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.615876 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:58.721532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.721428 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws5kh\" (UniqueName: \"kubernetes.io/projected/c69bca4f-dbbf-404d-9c2e-401300cee4b5-kube-api-access-ws5kh\") pod \"payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph\" (UID: \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:37:58.721917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.721697 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bca4f-dbbf-404d-9c2e-401300cee4b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph\" (UID: \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:37:58.745058 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.745015 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" event={"ID":"fc746f8f-fe26-4bd8-99da-cee03ed4f520","Type":"ContainerStarted","Data":"1d0f51bc566dc0154c499157fbf39af2bb79b08a0d865053704fe26e1fb0351c"} Oct 31 00:37:58.745230 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.745082 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" event={"ID":"fc746f8f-fe26-4bd8-99da-cee03ed4f520","Type":"ContainerStarted","Data":"87ecbed0fbeddf1171a8ec3a10913d51d0b45cd601063ab396fb608707b9c62c"} Oct 31 00:37:58.745333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.745310 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:37:58.746869 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.746839 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" event={"ID":"440f07dc-e69f-4e51-b1de-178d98232462","Type":"ContainerStarted","Data":"df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff"} Oct 31 00:37:58.747070 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.746938 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:37:58.748405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.748380 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" event={"ID":"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4","Type":"ContainerStarted","Data":"8ae7d6522695890cc122f54652f0a6d715101cacc2028e0a94c800c67a778ef4"} Oct 31 00:37:58.748515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.748420 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" event={"ID":"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4","Type":"ContainerStarted","Data":"ef587facbc1e9fe9625aa0c1c6a5cce3b01c0d0b0bc120b743dd7810377cbe43"} Oct 31 00:37:58.748703 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.748684 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:37:58.781724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.781669 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" podStartSLOduration=1.7816541510000001 podStartE2EDuration="1.781654151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:58.780721445 +0000 UTC m=+1096.478835210" watchObservedRunningTime="2026-10-31 00:37:58.781654151 +0000 UTC m=+1096.479767915" Oct 31 00:37:58.818310 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.818257 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" podStartSLOduration=3.172806419 podStartE2EDuration="5.818243144s" totalImagesPullingTime="2.645436725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:58.817421446 +0000 UTC m=+1096.515535432" watchObservedRunningTime="2026-10-31 00:37:58.818243144 +0000 UTC m=+1096.516356908" Oct 31 00:37:58.823289 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.823254 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bca4f-dbbf-404d-9c2e-401300cee4b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph\" (UID: \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:37:58.823289 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.823301 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws5kh\" (UniqueName: \"kubernetes.io/projected/c69bca4f-dbbf-404d-9c2e-401300cee4b5-kube-api-access-ws5kh\") pod \"payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph\" (UID: \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:37:58.824088 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.824065 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bca4f-dbbf-404d-9c2e-401300cee4b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph\" (UID: \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:37:58.844405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.844373 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws5kh\" (UniqueName: \"kubernetes.io/projected/c69bca4f-dbbf-404d-9c2e-401300cee4b5-kube-api-access-ws5kh\") pod \"payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph\" (UID: \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:37:58.852564 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.852511 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" podStartSLOduration=1.8524964179999999 podStartE2EDuration="1.852496418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:58.852181219 +0000 UTC m=+1096.550295009" watchObservedRunningTime="2026-10-31 00:37:58.852496418 +0000 UTC m=+1096.550610184" Oct 31 00:37:58.886661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:58.886625 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:37:59.183701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:59.183667 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph"] Oct 31 00:37:59.185836 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:37:59.185798 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69bca4f_dbbf_404d_9c2e_401300cee4b5.slice/crio-e76217e58452df51989250e3f2f354ee7f303ce47fca704dc50afc65b4c1399f WatchSource:0}: Error finding container e76217e58452df51989250e3f2f354ee7f303ce47fca704dc50afc65b4c1399f: Status 404 returned error can't find the container with id e76217e58452df51989250e3f2f354ee7f303ce47fca704dc50afc65b4c1399f Oct 31 00:37:59.756222 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:59.756181 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" event={"ID":"c69bca4f-dbbf-404d-9c2e-401300cee4b5","Type":"ContainerStarted","Data":"30bac281023632bbc61597532466cdd0f54de1b88623da7014b3c1b9e7883e19"} Oct 31 00:37:59.756668 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:59.756247 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" event={"ID":"c69bca4f-dbbf-404d-9c2e-401300cee4b5","Type":"ContainerStarted","Data":"e76217e58452df51989250e3f2f354ee7f303ce47fca704dc50afc65b4c1399f"} Oct 31 00:37:59.756668 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:59.756325 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:37:59.796426 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:37:59.796366 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" podStartSLOduration=1.796350307 podStartE2EDuration="1.796350307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:59.794605683 +0000 UTC m=+1097.492719448" watchObservedRunningTime="2026-10-31 00:37:59.796350307 +0000 UTC m=+1097.494464071" Oct 31 00:38:00.201434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.201396 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:00.366638 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.366592 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6"] Oct 31 00:38:00.370619 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.370594 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:00.376167 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.376139 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:00.385221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.385195 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-671959\"" Oct 31 00:38:00.392631 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.392584 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6"] Oct 31 00:38:00.428378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.428346 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm"] Oct 31 00:38:00.432182 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.432159 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:00.463417 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.463322 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm"] Oct 31 00:38:00.480123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.480091 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-671959-dockercfg-vcwml\"" Oct 31 00:38:00.543010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.542976 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7j2\" (UniqueName: \"kubernetes.io/projected/004ef2cb-1db6-4a6e-90af-70c614728a54-kube-api-access-hq7j2\") pod \"payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6\" (UID: \"004ef2cb-1db6-4a6e-90af-70c614728a54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:00.543010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.543018 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664ss\" (UniqueName: \"kubernetes.io/projected/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-kube-api-access-664ss\") pod \"payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm\" (UID: \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:00.543244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.543143 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm\" (UID: \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:00.543244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.543220 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004ef2cb-1db6-4a6e-90af-70c614728a54-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6\" (UID: \"004ef2cb-1db6-4a6e-90af-70c614728a54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:00.557899 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.557860 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:00.644344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.644308 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7j2\" (UniqueName: \"kubernetes.io/projected/004ef2cb-1db6-4a6e-90af-70c614728a54-kube-api-access-hq7j2\") pod \"payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6\" (UID: \"004ef2cb-1db6-4a6e-90af-70c614728a54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:00.644574 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.644362 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664ss\" (UniqueName: \"kubernetes.io/projected/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-kube-api-access-664ss\") pod \"payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm\" (UID: \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:00.644574 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.644438 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm\" (UID: \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:00.644574 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.644546 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004ef2cb-1db6-4a6e-90af-70c614728a54-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6\" (UID: \"004ef2cb-1db6-4a6e-90af-70c614728a54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:00.645255 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.645233 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004ef2cb-1db6-4a6e-90af-70c614728a54-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6\" (UID: \"004ef2cb-1db6-4a6e-90af-70c614728a54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:00.645294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.645262 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm\" (UID: \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:00.665759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.665729 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-664ss\" (UniqueName: \"kubernetes.io/projected/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-kube-api-access-664ss\") pod \"payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm\" (UID: \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:00.665910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.665886 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7j2\" (UniqueName: \"kubernetes.io/projected/004ef2cb-1db6-4a6e-90af-70c614728a54-kube-api-access-hq7j2\") pod \"payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6\" (UID: \"004ef2cb-1db6-4a6e-90af-70c614728a54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:00.685586 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.685548 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:00.706749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.706713 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:00.748936 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.748666 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:00.919178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.919133 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:00.982824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:00.982797 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6"] Oct 31 00:38:00.984709 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:38:00.984680 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004ef2cb_1db6_4a6e_90af_70c614728a54.slice/crio-1543f2f32c84970345abb73c0f8e0807e6bf6f033991f8185650fb71b9baf980 WatchSource:0}: Error finding container 1543f2f32c84970345abb73c0f8e0807e6bf6f033991f8185650fb71b9baf980: Status 404 returned error can't find the container with id 1543f2f32c84970345abb73c0f8e0807e6bf6f033991f8185650fb71b9baf980 Oct 31 00:38:01.054791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:01.054764 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm"] Oct 31 00:38:01.056852 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:38:01.056827 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e88831_aaa6_4c0c_96b1_4ab3b07754cc.slice/crio-beffc5d6f1a99eeedd3ca910d9ab2e5922436e74a4ac617bbb68b74f3d510d8c WatchSource:0}: Error finding container beffc5d6f1a99eeedd3ca910d9ab2e5922436e74a4ac617bbb68b74f3d510d8c: Status 404 returned error can't find the container with id beffc5d6f1a99eeedd3ca910d9ab2e5922436e74a4ac617bbb68b74f3d510d8c Oct 31 00:38:01.778807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:01.778771 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" event={"ID":"004ef2cb-1db6-4a6e-90af-70c614728a54","Type":"ContainerStarted","Data":"86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916"} Oct 31 00:38:01.779060 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:01.778975 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:01.779060 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:01.779027 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" event={"ID":"004ef2cb-1db6-4a6e-90af-70c614728a54","Type":"ContainerStarted","Data":"1543f2f32c84970345abb73c0f8e0807e6bf6f033991f8185650fb71b9baf980"} Oct 31 00:38:01.780640 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:01.780615 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" event={"ID":"44e88831-aaa6-4c0c-96b1-4ab3b07754cc","Type":"ContainerStarted","Data":"daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e"} Oct 31 00:38:01.780750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:01.780655 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" event={"ID":"44e88831-aaa6-4c0c-96b1-4ab3b07754cc","Type":"ContainerStarted","Data":"beffc5d6f1a99eeedd3ca910d9ab2e5922436e74a4ac617bbb68b74f3d510d8c"} Oct 31 00:38:01.780832 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:01.780754 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:01.881429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:01.881358 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" podStartSLOduration=1.881334066 podStartE2EDuration="1.881334066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:01.842894117 +0000 UTC m=+1099.541007884" watchObservedRunningTime="2026-10-31 00:38:01.881334066 +0000 UTC m=+1099.579447848" Oct 31 00:38:01.883046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:01.882994 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" podStartSLOduration=1.882977906 podStartE2EDuration="1.882977906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:01.879474642 +0000 UTC m=+1099.577588408" watchObservedRunningTime="2026-10-31 00:38:01.882977906 +0000 UTC m=+1099.581091671" Oct 31 00:38:02.305512 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.305446 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:02.446931 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.446900 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w"] Oct 31 00:38:02.451187 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.451159 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:02.466347 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.466302 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0d8bae-dockercfg-zzrr5\"" Oct 31 00:38:02.471545 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.470886 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w"] Oct 31 00:38:02.475080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.475048 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:02.475274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.475171 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0d8bae\"" Oct 31 00:38:02.479751 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.479719 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6xs7\" (UniqueName: \"kubernetes.io/projected/247397bb-656a-4a0d-b63f-02a7f742e04f-kube-api-access-m6xs7\") pod \"payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w\" (UID: \"247397bb-656a-4a0d-b63f-02a7f742e04f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:02.479996 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.479974 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247397bb-656a-4a0d-b63f-02a7f742e04f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w\" (UID: \"247397bb-656a-4a0d-b63f-02a7f742e04f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:02.549023 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.548991 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9"] Oct 31 00:38:02.553698 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.553674 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:02.585035 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.585004 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf314e-6c50-426e-9c41-ba911bea7b61-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9\" (UID: \"addf314e-6c50-426e-9c41-ba911bea7b61\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:02.585310 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.585291 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6xs7\" (UniqueName: \"kubernetes.io/projected/247397bb-656a-4a0d-b63f-02a7f742e04f-kube-api-access-m6xs7\") pod \"payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w\" (UID: \"247397bb-656a-4a0d-b63f-02a7f742e04f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:02.585442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.585427 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247397bb-656a-4a0d-b63f-02a7f742e04f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w\" (UID: \"247397bb-656a-4a0d-b63f-02a7f742e04f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:02.585695 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.585677 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bp4p\" (UniqueName: \"kubernetes.io/projected/addf314e-6c50-426e-9c41-ba911bea7b61-kube-api-access-8bp4p\") pod \"payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9\" (UID: \"addf314e-6c50-426e-9c41-ba911bea7b61\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:02.586646 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.586623 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247397bb-656a-4a0d-b63f-02a7f742e04f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w\" (UID: \"247397bb-656a-4a0d-b63f-02a7f742e04f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:02.600268 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.600232 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9"] Oct 31 00:38:02.643806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.643738 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6xs7\" (UniqueName: \"kubernetes.io/projected/247397bb-656a-4a0d-b63f-02a7f742e04f-kube-api-access-m6xs7\") pod \"payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w\" (UID: \"247397bb-656a-4a0d-b63f-02a7f742e04f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:02.687155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.687111 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf314e-6c50-426e-9c41-ba911bea7b61-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9\" (UID: \"addf314e-6c50-426e-9c41-ba911bea7b61\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:02.687343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.687294 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bp4p\" (UniqueName: \"kubernetes.io/projected/addf314e-6c50-426e-9c41-ba911bea7b61-kube-api-access-8bp4p\") pod \"payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9\" (UID: \"addf314e-6c50-426e-9c41-ba911bea7b61\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:02.687978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.687953 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf314e-6c50-426e-9c41-ba911bea7b61-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9\" (UID: \"addf314e-6c50-426e-9c41-ba911bea7b61\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:02.739354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.739219 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bp4p\" (UniqueName: \"kubernetes.io/projected/addf314e-6c50-426e-9c41-ba911bea7b61-kube-api-access-8bp4p\") pod \"payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9\" (UID: \"addf314e-6c50-426e-9c41-ba911bea7b61\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:02.746683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.746640 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:02.765413 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.765380 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:02.884504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.884473 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:02.896718 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:02.896683 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:03.041927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.041887 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:03.098804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.098773 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w"] Oct 31 00:38:03.100611 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:38:03.100547 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247397bb_656a_4a0d_b63f_02a7f742e04f.slice/crio-f72f4575c52533ea5c5200b7aba54dfc5e00f1a2ba1a07ddb4bcaee55e87119c WatchSource:0}: Error finding container f72f4575c52533ea5c5200b7aba54dfc5e00f1a2ba1a07ddb4bcaee55e87119c: Status 404 returned error can't find the container with id f72f4575c52533ea5c5200b7aba54dfc5e00f1a2ba1a07ddb4bcaee55e87119c Oct 31 00:38:03.187626 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:38:03.187596 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:38:03.241676 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.241624 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9"] Oct 31 00:38:03.242625 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:38:03.242592 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddf314e_6c50_426e_9c41_ba911bea7b61.slice/crio-f960d776a3125f869d1fe502b739e7c56f4be970afd1803ba8e470b8b00a6b0d WatchSource:0}: Error finding container f960d776a3125f869d1fe502b739e7c56f4be970afd1803ba8e470b8b00a6b0d: Status 404 returned error can't find the container with id f960d776a3125f869d1fe502b739e7c56f4be970afd1803ba8e470b8b00a6b0d Oct 31 00:38:03.795462 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.795407 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" event={"ID":"247397bb-656a-4a0d-b63f-02a7f742e04f","Type":"ContainerStarted","Data":"e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6"} Oct 31 00:38:03.795905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.795488 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" event={"ID":"247397bb-656a-4a0d-b63f-02a7f742e04f","Type":"ContainerStarted","Data":"f72f4575c52533ea5c5200b7aba54dfc5e00f1a2ba1a07ddb4bcaee55e87119c"} Oct 31 00:38:03.795905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.795615 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:03.797028 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.797001 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" event={"ID":"addf314e-6c50-426e-9c41-ba911bea7b61","Type":"ContainerStarted","Data":"1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636"} Oct 31 00:38:03.797142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.797050 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" event={"ID":"addf314e-6c50-426e-9c41-ba911bea7b61","Type":"ContainerStarted","Data":"f960d776a3125f869d1fe502b739e7c56f4be970afd1803ba8e470b8b00a6b0d"} Oct 31 00:38:03.797142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.797079 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:03.880430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.880375 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" podStartSLOduration=1.8803596420000002 podStartE2EDuration="1.880359642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.879014194 +0000 UTC m=+1101.577127977" watchObservedRunningTime="2026-10-31 00:38:03.880359642 +0000 UTC m=+1101.578473401" Oct 31 00:38:03.881313 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:03.881272 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" podStartSLOduration=1.881261913 podStartE2EDuration="1.881261913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.838673785 +0000 UTC m=+1101.536787551" watchObservedRunningTime="2026-10-31 00:38:03.881261913 +0000 UTC m=+1101.579375741" Oct 31 00:38:04.690171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:04.690135 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 00:38:04.690356 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:04.690232 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 00:38:04.737350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:04.737312 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:04.762010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:04.761980 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 00:38:04.881735 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:04.881703 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:05.062872 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:05.062764 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:06.736230 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:06.736196 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 00:38:06.736968 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:06.736943 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 00:38:06.803293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:06.803258 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:08.703410 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:08.703373 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:08.752933 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:08.752897 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 00:38:09.766755 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:09.766568 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 00:38:09.766755 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:09.766657 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 00:38:09.767292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:09.767140 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 00:38:10.772677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:10.771942 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:10.773867 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:10.773706 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 00:38:12.790621 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:12.790120 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 00:38:12.836724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:12.790688 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 00:38:12.996664 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:12.996623 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:13.990241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:13.990208 5978 generic.go:356] "Generic (PLEG): container finished" podID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerID="df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb" exitCode=0 Oct 31 00:38:13.990912 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:13.990889 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" event={"ID":"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6","Type":"ContainerDied","Data":"df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb"} Oct 31 00:38:13.991385 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:13.991369 5978 scope.go:122] "RemoveContainer" containerID="df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb" Oct 31 00:38:13.993219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:13.993178 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:13.993323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:13.993280 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:13.993378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:13.993361 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:14.807407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:14.807329 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 00:38:14.807407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:14.807391 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 00:38:15.002480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:15.002423 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" event={"ID":"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6","Type":"ContainerStarted","Data":"d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea"} Oct 31 00:38:15.239476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:15.238549 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:15.391244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:15.387801 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:15.662655 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:15.662617 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:38:15.662905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:15.662879 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:38:15.662978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:15.662938 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:38:16.044221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:16.044066 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:16.197600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:16.197527 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:16.368100 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:16.368004 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:16.569747 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:16.569707 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:16.663100 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:16.663009 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:38:16.663100 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:16.663076 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:38:16.757306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:16.757266 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:17.442645 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:17.442602 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:17.663519 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:17.663480 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:38:17.663728 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:17.663550 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:38:17.841547 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:17.841435 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:18.005728 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:18.005644 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:18.662888 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:18.662852 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:38:18.664258 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:18.662920 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:38:19.066827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:19.066796 5978 generic.go:356] "Generic (PLEG): container finished" podID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerID="f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d" exitCode=0 Oct 31 00:38:19.066990 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:19.066856 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" event={"ID":"65fa924f-1429-48ed-8ca2-bb5a3fca1f92","Type":"ContainerDied","Data":"f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d"} Oct 31 00:38:19.067242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:19.067229 5978 scope.go:122] "RemoveContainer" containerID="f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d" Oct 31 00:38:19.068588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:19.068556 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:19.068653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:19.068635 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:19.068688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:19.068672 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:19.656070 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:19.655986 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:38:19.663628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:19.663593 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:38:19.664016 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:19.663656 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:38:20.071340 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.071310 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:20.076186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.076141 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" event={"ID":"65fa924f-1429-48ed-8ca2-bb5a3fca1f92","Type":"ContainerStarted","Data":"f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72"} Oct 31 00:38:20.078258 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.078230 5978 generic.go:356] "Generic (PLEG): container finished" podID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerID="750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4" exitCode=0 Oct 31 00:38:20.078430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.078317 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" event={"ID":"a7ed5b38-29a4-42e8-933f-c7b305641872","Type":"ContainerDied","Data":"750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4"} Oct 31 00:38:20.078706 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.078687 5978 scope.go:122] "RemoveContainer" containerID="750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4" Oct 31 00:38:20.082520 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.082483 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:20.082648 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.082561 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:20.082648 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.082599 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:20.496200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.496104 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:38:20.497888 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.497858 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:38:20.498037 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.497920 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:38:20.660980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.660941 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:38:20.661168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.661018 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:38:20.952480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:20.944756 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:38:21.088926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:21.088374 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" event={"ID":"a7ed5b38-29a4-42e8-933f-c7b305641872","Type":"ContainerStarted","Data":"ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891"} Oct 31 00:38:21.089250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:21.089234 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 00:38:21.496495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:21.496123 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:38:21.496495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:21.496180 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:38:21.851480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:21.843220 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:38:21.851480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:21.844633 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:38:21.851480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:21.844689 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 00:38:21.918479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:21.916381 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.117396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:22.117250 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.277139 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:22.277107 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.496969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:22.496880 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:38:22.497232 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:22.497199 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:38:22.567042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:22.567009 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.835795 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:22.835755 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.843156 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:22.843111 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:38:22.843483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:22.843418 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 00:38:23.039119 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:23.039086 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:23.496218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:23.496183 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:38:23.496665 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:23.496234 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:38:23.840989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:23.840947 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:38:23.841409 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:23.841380 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 00:38:24.496622 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:24.496588 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:38:24.497127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:24.497102 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:38:24.500865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:24.500840 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:24.840022 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:24.839493 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:38:24.840022 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:24.839549 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 00:38:25.496405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:25.496364 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:38:25.496625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:25.496424 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:38:25.843038 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:25.842997 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:38:25.843704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:25.843663 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 00:38:26.420302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:26.418789 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:26.500390 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:26.500356 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:38:26.500782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:26.500759 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:38:26.501551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:26.501530 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 00:38:26.898741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:26.898703 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:38:26.899659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:26.899638 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:38:26.900439 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:26.900418 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:38:27.035859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:27.035830 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:27.193880 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:27.193790 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:27.710790 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:27.710748 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:27.878918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:27.878883 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:28.039416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:28.039322 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:28.189028 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:28.188995 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:29.807609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:29.807565 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:29.962254 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:29.962205 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:31.865199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:31.865165 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:33.182137 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:38:33.182098 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:38:33.615030 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:33.614978 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:35.295506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:35.295470 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:35.550332 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:35.550240 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:37.007901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:37.007836 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:38.620851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:38.620807 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:38.778018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:38.777975 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:40.291466 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:40.291419 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:42.086190 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:42.086120 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:43.645204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:43.645168 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:43.782472 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:43.782428 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:44.862865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:44.862830 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd"] Oct 31 00:38:44.868921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:44.868892 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:44.884999 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:44.884952 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"openshift-service-ca.crt\"" Oct 31 00:38:44.891917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:44.891889 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"kube-root-ca.crt\"" Oct 31 00:38:44.892066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:44.891889 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"simulator-w0-671959-kserve-self-signed-certs\"" Oct 31 00:38:44.892066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:44.891889 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"default-dockercfg-m5n9g\"" Oct 31 00:38:44.897588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:44.897561 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd"] Oct 31 00:38:45.001978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.001886 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73fb67-5047-4046-b0f4-c647394f2f05-tls-certs\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.001978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.001944 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-home\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.002181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.002037 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-model-cache\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.002181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.002133 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twggq\" (UniqueName: \"kubernetes.io/projected/bf73fb67-5047-4046-b0f4-c647394f2f05-kube-api-access-twggq\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.002181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.002170 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-dshm\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.002277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.002237 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-tmp-dir\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.103109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.103072 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-model-cache\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.103323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.103136 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twggq\" (UniqueName: \"kubernetes.io/projected/bf73fb67-5047-4046-b0f4-c647394f2f05-kube-api-access-twggq\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.103323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.103164 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-dshm\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.103323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.103193 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-tmp-dir\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.103323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.103280 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73fb67-5047-4046-b0f4-c647394f2f05-tls-certs\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.103323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.103316 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-home\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.103670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.103575 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-model-cache\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.103742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.103685 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-home\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.103817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.103800 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-tmp-dir\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.105970 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.105942 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-dshm\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.106537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.106518 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73fb67-5047-4046-b0f4-c647394f2f05-tls-certs\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.120117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.120085 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:45.125074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.125051 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twggq\" (UniqueName: \"kubernetes.io/projected/bf73fb67-5047-4046-b0f4-c647394f2f05-kube-api-access-twggq\") pod \"simulator-w0-671959-kserve-74b4fcb8b-2gdvd\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.184346 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.184314 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:45.304652 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.304571 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:45.498102 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:45.498071 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd"] Oct 31 00:38:45.502294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:38:45.502249 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf73fb67_5047_4046_b0f4_c647394f2f05.slice/crio-6211ea78eec3f58808ae9391d3e6380ed807a2d95a834badd62e96572a90ea71 WatchSource:0}: Error finding container 6211ea78eec3f58808ae9391d3e6380ed807a2d95a834badd62e96572a90ea71: Status 404 returned error can't find the container with id 6211ea78eec3f58808ae9391d3e6380ed807a2d95a834badd62e96572a90ea71 Oct 31 00:38:46.271003 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:46.270957 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" event={"ID":"bf73fb67-5047-4046-b0f4-c647394f2f05","Type":"ContainerStarted","Data":"a927de4b0c29935075257f4e70d6f358e092156dd78ac85bcc1fb8044da326b6"} Oct 31 00:38:46.271404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:46.271012 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" event={"ID":"bf73fb67-5047-4046-b0f4-c647394f2f05","Type":"ContainerStarted","Data":"6211ea78eec3f58808ae9391d3e6380ed807a2d95a834badd62e96572a90ea71"} Oct 31 00:38:46.311865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:46.311811 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podStartSLOduration=2.311797545 podStartE2EDuration="2.311797545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:46.308229385 +0000 UTC m=+1144.006343150" watchObservedRunningTime="2026-10-31 00:38:46.311797545 +0000 UTC m=+1144.009911309" Oct 31 00:38:46.976163 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:46.976124 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:47.185570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:47.185532 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:48.391974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:48.391929 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:48.678296 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:48.678203 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:50.393320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:50.393285 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:51.996091 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:51.996051 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:53.814107 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:53.814055 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:55.185322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:55.185265 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:55.199787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:55.199757 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:55.338616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:55.338584 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:38:55.411988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:55.411951 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:56.237060 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:56.237027 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:56.992888 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:56.992806 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:58.592668 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:38:58.592630 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:00.297624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:00.297581 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:02.146013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:02.145968 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:03.109109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.109069 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc"] Oct 31 00:39:03.113420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.113392 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.136017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.135986 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"default-dockercfg-fhc67\"" Oct 31 00:39:03.136242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.136065 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"kube-root-ca.crt\"" Oct 31 00:39:03.136334 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.136065 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"openshift-service-ca.crt\"" Oct 31 00:39:03.136334 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.136162 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-daaf26\"/\"premium-w3-daaf26-kserve-self-signed-certs\"" Oct 31 00:39:03.138839 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.138812 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc"] Oct 31 00:39:03.181490 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:39:03.181438 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:39:03.217327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.217287 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:03.226026 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.225998 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tls-certs\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.226200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.226062 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tmp-dir\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.226200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.226152 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-home\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.226200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.226195 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-dshm\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.226373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.226355 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-model-cache\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.226409 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.226396 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76g6\" (UniqueName: \"kubernetes.io/projected/02f2f37e-6172-4c79-8973-cfdf4c660bfd-kube-api-access-s76g6\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.329250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.329214 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-model-cache\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.329505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.329271 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76g6\" (UniqueName: \"kubernetes.io/projected/02f2f37e-6172-4c79-8973-cfdf4c660bfd-kube-api-access-s76g6\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.329505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.329305 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tls-certs\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.329505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.329370 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tmp-dir\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.329505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.329407 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-home\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.329505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.329473 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-dshm\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.329785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.329758 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-model-cache\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.330240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.330210 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tmp-dir\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.330735 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.330683 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-home\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.332798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.332772 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-dshm\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.333411 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.333388 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tls-certs\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.352494 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.352464 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76g6\" (UniqueName: \"kubernetes.io/projected/02f2f37e-6172-4c79-8973-cfdf4c660bfd-kube-api-access-s76g6\") pod \"premium-w3-daaf26-kserve-7f4475df7f-wmxlc\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.430253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.430155 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:03.728260 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.728232 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc"] Oct 31 00:39:03.730548 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:39:03.730519 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02f2f37e_6172_4c79_8973_cfdf4c660bfd.slice/crio-c378d1eeb706cf165533bd028c0b41e99c696749ad5e9768bd9b977346b13e6f WatchSource:0}: Error finding container c378d1eeb706cf165533bd028c0b41e99c696749ad5e9768bd9b977346b13e6f: Status 404 returned error can't find the container with id c378d1eeb706cf165533bd028c0b41e99c696749ad5e9768bd9b977346b13e6f Oct 31 00:39:03.889308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:03.889278 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:04.399444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:04.399406 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" event={"ID":"02f2f37e-6172-4c79-8973-cfdf4c660bfd","Type":"ContainerStarted","Data":"73703e0dc337d928d4915e9da5a9e293d9e441edbc525f9d3f1ea7190c4729d4"} Oct 31 00:39:04.399879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:04.399481 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" event={"ID":"02f2f37e-6172-4c79-8973-cfdf4c660bfd","Type":"ContainerStarted","Data":"c378d1eeb706cf165533bd028c0b41e99c696749ad5e9768bd9b977346b13e6f"} Oct 31 00:39:04.431978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:04.431939 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:04.441004 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:04.440949 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podStartSLOduration=1.440933778 podStartE2EDuration="1.440933778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:04.439293989 +0000 UTC m=+1162.137407747" watchObservedRunningTime="2026-10-31 00:39:04.440933778 +0000 UTC m=+1162.139047543" Oct 31 00:39:05.372770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:05.372735 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:05.526292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:05.526249 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:07.209206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:07.209168 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:09.002057 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:09.002000 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:10.688582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:10.688526 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:12.201503 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:12.201468 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:12.455789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:12.455703 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:13.431000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:13.430960 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:13.443490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:13.443449 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:13.464301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:13.464268 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:39:13.840171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:13.840136 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:15.548517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:15.548477 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:17.150585 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:17.150546 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:18.791797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:18.791753 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:18.941055 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:18.941011 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:20.392333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:20.392285 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:22.047976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.047941 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl"] Oct 31 00:39:22.053856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.053832 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.071770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.071745 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"premium-w6-be301b-kserve-self-signed-certs\"" Oct 31 00:39:22.081818 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.081787 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"openshift-service-ca.crt\"" Oct 31 00:39:22.081994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.081834 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"kube-root-ca.crt\"" Oct 31 00:39:22.081994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.081931 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be301b\"/\"default-dockercfg-bqtkx\"" Oct 31 00:39:22.085830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.085802 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl"] Oct 31 00:39:22.150648 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.150595 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:22.152676 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.152657 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-home\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.152787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.152685 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tls-certs\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.152787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.152764 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-model-cache\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.152862 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.152833 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-dshm\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.152964 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.152951 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tmp-dir\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.153014 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.152998 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx54d\" (UniqueName: \"kubernetes.io/projected/501ba3a7-cbea-4081-bf2f-e19c6da9d390-kube-api-access-zx54d\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.254128 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.254089 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-dshm\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.254520 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.254490 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tmp-dir\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.254740 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.254720 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx54d\" (UniqueName: \"kubernetes.io/projected/501ba3a7-cbea-4081-bf2f-e19c6da9d390-kube-api-access-zx54d\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.255214 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.255194 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-home\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.256827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.256808 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tls-certs\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.257598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.257580 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-model-cache\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.257823 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.256734 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-home\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.257944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.254917 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tmp-dir\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.258045 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.257973 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-model-cache\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.261894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.261845 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tls-certs\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.263233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.263214 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-dshm\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.277002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.276945 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx54d\" (UniqueName: \"kubernetes.io/projected/501ba3a7-cbea-4081-bf2f-e19c6da9d390-kube-api-access-zx54d\") pod \"premium-w6-be301b-kserve-77b87789f9-mbgnl\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.295007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.294969 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:22.372088 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.372052 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:22.668983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:22.668950 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl"] Oct 31 00:39:22.670608 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:39:22.670580 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod501ba3a7_cbea_4081_bf2f_e19c6da9d390.slice/crio-e239b6a73be7a157a3f1b65e929348cf1866b4f4a14dda48116e6a66db451b3e WatchSource:0}: Error finding container e239b6a73be7a157a3f1b65e929348cf1866b4f4a14dda48116e6a66db451b3e: Status 404 returned error can't find the container with id e239b6a73be7a157a3f1b65e929348cf1866b4f4a14dda48116e6a66db451b3e Oct 31 00:39:23.516918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:23.516882 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" event={"ID":"501ba3a7-cbea-4081-bf2f-e19c6da9d390","Type":"ContainerStarted","Data":"ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7"} Oct 31 00:39:23.516918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:23.516928 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" event={"ID":"501ba3a7-cbea-4081-bf2f-e19c6da9d390","Type":"ContainerStarted","Data":"e239b6a73be7a157a3f1b65e929348cf1866b4f4a14dda48116e6a66db451b3e"} Oct 31 00:39:23.556387 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:23.556333 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podStartSLOduration=1.556317876 podStartE2EDuration="1.556317876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:23.553346493 +0000 UTC m=+1181.251460291" watchObservedRunningTime="2026-10-31 00:39:23.556317876 +0000 UTC m=+1181.254431640" Oct 31 00:39:23.823660 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:23.823623 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:24.372544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:24.372504 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:25.483712 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.483670 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:25.620540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.620509 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z"] Oct 31 00:39:25.632894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.632870 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.644368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.644339 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z"] Oct 31 00:39:25.651796 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.651771 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"premium-w0-671959-kserve-self-signed-certs\"" Oct 31 00:39:25.710205 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.710174 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:25.800924 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.800832 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56787030-da65-4d0a-93ce-41ed20e64b19-tls-certs\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.801103 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.800979 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-tmp-dir\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.801103 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.801041 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvf5\" (UniqueName: \"kubernetes.io/projected/56787030-da65-4d0a-93ce-41ed20e64b19-kube-api-access-kmvf5\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.801103 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.801067 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-model-cache\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.801243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.801223 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-dshm\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.801324 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.801311 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-home\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.902261 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.902223 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-dshm\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.902448 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.902282 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-home\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.902448 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.902334 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56787030-da65-4d0a-93ce-41ed20e64b19-tls-certs\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.902448 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.902408 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-tmp-dir\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.902670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.902492 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvf5\" (UniqueName: \"kubernetes.io/projected/56787030-da65-4d0a-93ce-41ed20e64b19-kube-api-access-kmvf5\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.902670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.902534 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-model-cache\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.902814 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.902791 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-home\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.902872 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.902841 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-tmp-dir\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.902913 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.902886 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-model-cache\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.904667 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.904643 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-dshm\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.904822 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.904804 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56787030-da65-4d0a-93ce-41ed20e64b19-tls-certs\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.923530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.923500 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvf5\" (UniqueName: \"kubernetes.io/projected/56787030-da65-4d0a-93ce-41ed20e64b19-kube-api-access-kmvf5\") pod \"premium-w0-671959-kserve-5fc47f4fcf-2c48z\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:25.947075 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:25.947046 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:26.244080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:26.244052 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z"] Oct 31 00:39:26.245413 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:39:26.245384 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56787030_da65_4d0a_93ce_41ed20e64b19.slice/crio-fa62924d2d6cb3b3af3b4417fdf41094f09f76e6c8c1f4f66ac8556ace92cd04 WatchSource:0}: Error finding container fa62924d2d6cb3b3af3b4417fdf41094f09f76e6c8c1f4f66ac8556ace92cd04: Status 404 returned error can't find the container with id fa62924d2d6cb3b3af3b4417fdf41094f09f76e6c8c1f4f66ac8556ace92cd04 Oct 31 00:39:26.536748 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:26.536667 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" event={"ID":"56787030-da65-4d0a-93ce-41ed20e64b19","Type":"ContainerStarted","Data":"59eb9d63ad578af786838827064133eb305245956ad948fd219b1a2d63254483"} Oct 31 00:39:26.536748 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:26.536706 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" event={"ID":"56787030-da65-4d0a-93ce-41ed20e64b19","Type":"ContainerStarted","Data":"fa62924d2d6cb3b3af3b4417fdf41094f09f76e6c8c1f4f66ac8556ace92cd04"} Oct 31 00:39:26.579636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:26.579582 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podStartSLOduration=1.579568069 podStartE2EDuration="1.579568069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:26.57815201 +0000 UTC m=+1184.276265777" watchObservedRunningTime="2026-10-31 00:39:26.579568069 +0000 UTC m=+1184.277681833" Oct 31 00:39:26.948046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:26.948014 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:27.075944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:27.075912 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:28.103273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:28.103235 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:28.935419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:28.935386 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:30.616090 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:30.616052 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:32.113650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.113610 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5"] Oct 31 00:39:32.117511 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.117489 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.130213 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.130185 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-735123-openshift-default-dockercfg-z4chw\"" Oct 31 00:39:32.147877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.147845 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5"] Oct 31 00:39:32.228078 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.228041 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:32.287914 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.287883 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60da948-ebf0-4065-977a-a502ded06209-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.288097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.287946 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.288097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.288047 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.288097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.288063 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60da948-ebf0-4065-977a-a502ded06209-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.288208 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.288122 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.288242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.288206 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td2mw\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-kube-api-access-td2mw\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.288275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.288263 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.288310 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.288294 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.288404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.288390 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.373051 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.372968 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:32.386148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.386112 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:32.389839 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.389816 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.389926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.389852 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.389926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.389869 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60da948-ebf0-4065-977a-a502ded06209-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.389965 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390060 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.390045 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td2mw\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-kube-api-access-td2mw\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.390096 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.390144 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.390231 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.390270 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390375 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.390309 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.390425 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60da948-ebf0-4065-977a-a502ded06209-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390564 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.390539 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.390658 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.390633 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.391254 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.391234 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60da948-ebf0-4065-977a-a502ded06209-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.392302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.392280 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.392570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.392553 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60da948-ebf0-4065-977a-a502ded06209-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.411532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.411498 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.412617 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.412591 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td2mw\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-kube-api-access-td2mw\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.432658 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.432626 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:32.599288 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.599259 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:39:32.746848 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.746823 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5"] Oct 31 00:39:32.748882 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:39:32.748854 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60da948_ebf0_4065_977a_a502ded06209.slice/crio-bf250bd2cc9d5aa880e5af3543e4b195dead8190eae738e393391a8d4757b8bc WatchSource:0}: Error finding container bf250bd2cc9d5aa880e5af3543e4b195dead8190eae738e393391a8d4757b8bc: Status 404 returned error can't find the container with id bf250bd2cc9d5aa880e5af3543e4b195dead8190eae738e393391a8d4757b8bc Oct 31 00:39:32.752716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.752680 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:32.752886 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.752754 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:32.752886 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:32.752792 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:33.190965 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:39:33.190928 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:39:33.595363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:33.595330 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" event={"ID":"c60da948-ebf0-4065-977a-a502ded06209","Type":"ContainerStarted","Data":"da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29"} Oct 31 00:39:33.595562 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:33.595389 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" event={"ID":"c60da948-ebf0-4065-977a-a502ded06209","Type":"ContainerStarted","Data":"bf250bd2cc9d5aa880e5af3543e4b195dead8190eae738e393391a8d4757b8bc"} Oct 31 00:39:33.658587 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:33.658530 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" podStartSLOduration=1.6585152010000002 podStartE2EDuration="1.658515201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:33.657490462 +0000 UTC m=+1191.355604232" watchObservedRunningTime="2026-10-31 00:39:33.658515201 +0000 UTC m=+1191.356628960" Oct 31 00:39:33.995398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:33.995304 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:34.437582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:34.437550 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:39:34.438188 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:34.437600 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:39:34.442303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:34.442278 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:35.435912 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:35.435868 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:39:35.436101 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:35.435940 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:39:35.587524 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:35.587485 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:35.947203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:35.947166 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:35.960155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:35.960129 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:36.435850 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:36.435803 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:39:36.436065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:36.435865 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:39:36.648891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:36.648856 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:39:37.202688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:37.202571 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:37.437477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:37.435992 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:39:37.437784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:37.437752 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:39:38.437483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:38.436748 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:39:38.437483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:38.436809 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 00:39:38.896894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:38.896854 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:39.602013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:39.601683 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:39.603164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:39.603140 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:39.604804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:39.604780 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:39:40.592005 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:40.591967 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:42.294739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:42.294695 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:42.619954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:42.619922 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:39:42.623645 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:42.623623 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:39:43.772556 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:43.772522 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:44.003646 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:44.003608 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:45.608692 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:45.608612 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:47.140583 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.140548 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:47.456218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.456124 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp"] Oct 31 00:39:47.461726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.461694 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.475820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.475792 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c9e377-openshift-default-dockercfg-hqt4g\"" Oct 31 00:39:47.489967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.489932 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp"] Oct 31 00:39:47.512008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.511975 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.512190 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.512026 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdpnh\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-kube-api-access-pdpnh\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.512235 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.512196 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.512277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.512265 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cb6d77-6b91-4cb8-8205-16567a3783dd-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.512343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.512327 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.512388 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.512373 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.512517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.512494 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.512602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.512547 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.512662 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.512627 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614194 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614245 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cb6d77-6b91-4cb8-8205-16567a3783dd-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614285 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614314 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614335 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614373 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614740 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614625 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614795 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614768 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614795 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614776 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614907 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614832 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614907 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614843 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpnh\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-kube-api-access-pdpnh\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.614907 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.614898 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.615072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.615021 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.615213 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.615189 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cb6d77-6b91-4cb8-8205-16567a3783dd-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.616828 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.616809 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.617225 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.617205 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.635594 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.635565 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.635749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.635626 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdpnh\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-kube-api-access-pdpnh\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:47.776433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:47.776344 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:48.090758 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:48.090726 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp"] Oct 31 00:39:48.092294 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:39:48.092266 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cb6d77_6b91_4cb8_8205_16567a3783dd.slice/crio-f8e824a31ea047c07fdc8ec8ac98ffd35b0e01ac63ae7298df6e09683257ff0e WatchSource:0}: Error finding container f8e824a31ea047c07fdc8ec8ac98ffd35b0e01ac63ae7298df6e09683257ff0e: Status 404 returned error can't find the container with id f8e824a31ea047c07fdc8ec8ac98ffd35b0e01ac63ae7298df6e09683257ff0e Oct 31 00:39:48.095199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:48.095158 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:48.095285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:48.095256 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:48.095324 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:48.095307 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:39:48.801980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:48.801939 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" event={"ID":"04cb6d77-6b91-4cb8-8205-16567a3783dd","Type":"ContainerStarted","Data":"96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb"} Oct 31 00:39:48.802663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:48.802634 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" event={"ID":"04cb6d77-6b91-4cb8-8205-16567a3783dd","Type":"ContainerStarted","Data":"f8e824a31ea047c07fdc8ec8ac98ffd35b0e01ac63ae7298df6e09683257ff0e"} Oct 31 00:39:48.867986 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:48.867908 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" podStartSLOduration=1.867883322 podStartE2EDuration="1.867883322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:48.865224244 +0000 UTC m=+1206.563338010" watchObservedRunningTime="2026-10-31 00:39:48.867883322 +0000 UTC m=+1206.565997091" Oct 31 00:39:49.025733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:49.025679 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:49.776727 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:49.776685 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:49.778462 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:49.778429 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:39:49.778541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:49.778517 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:39:50.693498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:50.693439 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:50.777131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:50.777096 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:39:50.777307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:50.777154 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:39:51.777357 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:51.777323 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:39:51.777817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:51.777378 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:39:52.405735 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:52.405692 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:52.776988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:52.776903 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:39:52.776988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:52.776962 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:39:53.777278 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:53.777234 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:39:53.777705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:53.777305 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:39:54.235787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:54.235748 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:54.395195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:54.395136 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:54.778238 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:54.777783 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:39:54.778238 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:54.777863 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:39:55.781911 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:55.781876 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:55.782322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:55.782278 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:55.783048 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:55.783023 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:39:55.889005 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:55.888967 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:57.495413 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:57.495374 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:59.226512 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:39:59.226469 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:00.795068 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:00.795035 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:02.404242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:02.404199 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:03.325107 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:40:03.325039 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:40:04.239300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:04.239264 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:06.180440 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:06.180398 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:07.589831 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.589794 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x"] Oct 31 00:40:07.594184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.594163 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.607227 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.607199 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:07.617887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.617855 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:40:07.622302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.622220 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x"] Oct 31 00:40:07.681202 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.681165 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d28da1e6-7f59x\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.681392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.681348 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d28da1e6-7f59x\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.681392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.681376 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxzm\" (UniqueName: \"kubernetes.io/projected/d5393f4f-6f8f-488c-93ea-f209802265cd-kube-api-access-dsxzm\") pod \"maas-api-revoke-sub-d28da1e6-7f59x\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.783037 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.782987 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d28da1e6-7f59x\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.783037 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.783043 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxzm\" (UniqueName: \"kubernetes.io/projected/d5393f4f-6f8f-488c-93ea-f209802265cd-kube-api-access-dsxzm\") pod \"maas-api-revoke-sub-d28da1e6-7f59x\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.783320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.783205 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d28da1e6-7f59x\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.783981 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.783959 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d28da1e6-7f59x\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.787025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.786945 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d28da1e6-7f59x\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.823478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.817320 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxzm\" (UniqueName: \"kubernetes.io/projected/d5393f4f-6f8f-488c-93ea-f209802265cd-kube-api-access-dsxzm\") pod \"maas-api-revoke-sub-d28da1e6-7f59x\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:07.910514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:07.910389 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:08.210018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:08.209906 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:08.233475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:08.227117 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x"] Oct 31 00:40:08.270982 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:08.270956 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:40:08.966201 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:08.966158 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" event={"ID":"d5393f4f-6f8f-488c-93ea-f209802265cd","Type":"ContainerStarted","Data":"2ddad9268c61300600ba4051c46a582861b48e215ab63152e1e4278c52073679"} Oct 31 00:40:10.106409 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:10.106370 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:11.796072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:11.795977 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:11.989069 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:11.989031 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" event={"ID":"d5393f4f-6f8f-488c-93ea-f209802265cd","Type":"ContainerStarted","Data":"87b4e0f9694ac2e39db1cbebe68b0228980c52e032ec612b8aaf2bfe5f4ed201"} Oct 31 00:40:12.027107 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:12.027045 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" podStartSLOduration=2.041895241 podStartE2EDuration="5.027026172s" totalImagesPullingTime="2.985130931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:12.024679665 +0000 UTC m=+1229.722793430" watchObservedRunningTime="2026-10-31 00:40:12.027026172 +0000 UTC m=+1229.725139938" Oct 31 00:40:12.996890 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:12.996854 5978 generic.go:356] "Generic (PLEG): container finished" podID="d5393f4f-6f8f-488c-93ea-f209802265cd" containerID="87b4e0f9694ac2e39db1cbebe68b0228980c52e032ec612b8aaf2bfe5f4ed201" exitCode=0 Oct 31 00:40:12.997361 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:12.996943 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" event={"ID":"d5393f4f-6f8f-488c-93ea-f209802265cd","Type":"ContainerDied","Data":"87b4e0f9694ac2e39db1cbebe68b0228980c52e032ec612b8aaf2bfe5f4ed201"} Oct 31 00:40:13.479916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:13.479858 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:14.166939 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.166913 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:14.262393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.262303 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-xks-ca\") pod \"d5393f4f-6f8f-488c-93ea-f209802265cd\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " Oct 31 00:40:14.262393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.262353 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-openshift-ca\") pod \"d5393f4f-6f8f-488c-93ea-f209802265cd\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " Oct 31 00:40:14.262690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.262399 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5393f4f-6f8f-488c-93ea-f209802265cd-kube-api-access-dsxzm\" (UniqueName: \"kubernetes.io/projected/d5393f4f-6f8f-488c-93ea-f209802265cd-kube-api-access-dsxzm\") pod \"d5393f4f-6f8f-488c-93ea-f209802265cd\" (UID: \"d5393f4f-6f8f-488c-93ea-f209802265cd\") " Oct 31 00:40:14.262834 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.262811 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-openshift-ca" pod "d5393f4f-6f8f-488c-93ea-f209802265cd" (UID: "d5393f4f-6f8f-488c-93ea-f209802265cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:14.264756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.264729 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5393f4f-6f8f-488c-93ea-f209802265cd-kube-api-access-dsxzm" pod "d5393f4f-6f8f-488c-93ea-f209802265cd" (UID: "d5393f4f-6f8f-488c-93ea-f209802265cd"). InnerVolumeSpecName "kube-api-access-dsxzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:14.264860 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.264780 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-xks-ca" pod "d5393f4f-6f8f-488c-93ea-f209802265cd" (UID: "d5393f4f-6f8f-488c-93ea-f209802265cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:14.364419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.364379 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:14.364419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.364413 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5393f4f-6f8f-488c-93ea-f209802265cd-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:14.364419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:14.364423 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsxzm\" (UniqueName: \"kubernetes.io/projected/d5393f4f-6f8f-488c-93ea-f209802265cd-kube-api-access-dsxzm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:15.010689 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:15.010653 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" event={"ID":"d5393f4f-6f8f-488c-93ea-f209802265cd","Type":"ContainerDied","Data":"2ddad9268c61300600ba4051c46a582861b48e215ab63152e1e4278c52073679"} Oct 31 00:40:15.010875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:15.010704 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ddad9268c61300600ba4051c46a582861b48e215ab63152e1e4278c52073679" Oct 31 00:40:15.010875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:15.010667 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x" Oct 31 00:40:15.107833 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:15.107790 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:16.677195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:16.677163 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:18.498177 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:18.498133 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:20.155203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:20.155162 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:21.894709 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:21.894672 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:22.228864 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.228782 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw"] Oct 31 00:40:22.229489 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.229466 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d5393f4f-6f8f-488c-93ea-f209802265cd" containerName="revoke-keys" Oct 31 00:40:22.229489 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.229485 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5393f4f-6f8f-488c-93ea-f209802265cd" containerName="revoke-keys" Oct 31 00:40:22.229677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.229604 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5393f4f-6f8f-488c-93ea-f209802265cd" containerName="revoke-keys" Oct 31 00:40:22.233485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.233464 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.252819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.252788 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw"] Oct 31 00:40:22.256760 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.256720 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:22.256760 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.256758 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:40:22.409344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.409305 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hjp\" (UniqueName: \"kubernetes.io/projected/5f3ea689-f4a7-4b83-afcc-7446124457ae-kube-api-access-69hjp\") pod \"maas-api-revoke-sub-edc631a8-txxlw\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.409637 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.409509 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edc631a8-txxlw\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.409637 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.409552 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edc631a8-txxlw\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.511168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.511073 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hjp\" (UniqueName: \"kubernetes.io/projected/5f3ea689-f4a7-4b83-afcc-7446124457ae-kube-api-access-69hjp\") pod \"maas-api-revoke-sub-edc631a8-txxlw\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.511336 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.511218 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edc631a8-txxlw\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.511336 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.511249 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edc631a8-txxlw\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.512113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.512089 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edc631a8-txxlw\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.513716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.513691 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edc631a8-txxlw\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.532855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.532826 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hjp\" (UniqueName: \"kubernetes.io/projected/5f3ea689-f4a7-4b83-afcc-7446124457ae-kube-api-access-69hjp\") pod \"maas-api-revoke-sub-edc631a8-txxlw\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.547807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.547776 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:22.850711 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:22.849993 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw"] Oct 31 00:40:22.853238 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:40:22.853202 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3ea689_f4a7_4b83_afcc_7446124457ae.slice/crio-6b19377645de82221f38138f39b09e1b83342f4831336b194c0201783551144a WatchSource:0}: Error finding container 6b19377645de82221f38138f39b09e1b83342f4831336b194c0201783551144a: Status 404 returned error can't find the container with id 6b19377645de82221f38138f39b09e1b83342f4831336b194c0201783551144a Oct 31 00:40:23.115528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:23.115123 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" event={"ID":"5f3ea689-f4a7-4b83-afcc-7446124457ae","Type":"ContainerStarted","Data":"6b19377645de82221f38138f39b09e1b83342f4831336b194c0201783551144a"} Oct 31 00:40:23.598735 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:23.598690 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:23.743679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:23.743630 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:24.122744 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:24.122706 5978 generic.go:356] "Generic (PLEG): container finished" podID="5f3ea689-f4a7-4b83-afcc-7446124457ae" containerID="2480507d171a0ebb0055e1dc084b7e00664682b251c6fdc080f53d2c8b08c096" exitCode=0 Oct 31 00:40:24.123169 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:24.122792 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" event={"ID":"5f3ea689-f4a7-4b83-afcc-7446124457ae","Type":"ContainerDied","Data":"2480507d171a0ebb0055e1dc084b7e00664682b251c6fdc080f53d2c8b08c096"} Oct 31 00:40:25.191121 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.189927 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:25.289312 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.289277 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:25.460151 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.460062 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-xks-ca\") pod \"5f3ea689-f4a7-4b83-afcc-7446124457ae\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " Oct 31 00:40:25.460327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.460227 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-openshift-ca\") pod \"5f3ea689-f4a7-4b83-afcc-7446124457ae\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " Oct 31 00:40:25.460327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.460253 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3ea689-f4a7-4b83-afcc-7446124457ae-kube-api-access-69hjp\" (UniqueName: \"kubernetes.io/projected/5f3ea689-f4a7-4b83-afcc-7446124457ae-kube-api-access-69hjp\") pod \"5f3ea689-f4a7-4b83-afcc-7446124457ae\" (UID: \"5f3ea689-f4a7-4b83-afcc-7446124457ae\") " Oct 31 00:40:25.460697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.460668 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-openshift-ca" pod "5f3ea689-f4a7-4b83-afcc-7446124457ae" (UID: "5f3ea689-f4a7-4b83-afcc-7446124457ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:25.460893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.460842 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:25.462420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.462389 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-xks-ca" pod "5f3ea689-f4a7-4b83-afcc-7446124457ae" (UID: "5f3ea689-f4a7-4b83-afcc-7446124457ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:25.462593 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.462431 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3ea689-f4a7-4b83-afcc-7446124457ae-kube-api-access-69hjp" pod "5f3ea689-f4a7-4b83-afcc-7446124457ae" (UID: "5f3ea689-f4a7-4b83-afcc-7446124457ae"). InnerVolumeSpecName "kube-api-access-69hjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:25.562702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.562654 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hjp\" (UniqueName: \"kubernetes.io/projected/5f3ea689-f4a7-4b83-afcc-7446124457ae-kube-api-access-69hjp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:25.562885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:25.562775 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3ea689-f4a7-4b83-afcc-7446124457ae-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:26.137354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:26.137320 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" Oct 31 00:40:26.137354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:26.137346 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw" event={"ID":"5f3ea689-f4a7-4b83-afcc-7446124457ae","Type":"ContainerDied","Data":"6b19377645de82221f38138f39b09e1b83342f4831336b194c0201783551144a"} Oct 31 00:40:26.137693 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:26.137395 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b19377645de82221f38138f39b09e1b83342f4831336b194c0201783551144a" Oct 31 00:40:27.078919 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.078882 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:27.405247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.405154 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw"] Oct 31 00:40:27.405942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.405919 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5f3ea689-f4a7-4b83-afcc-7446124457ae" containerName="revoke-keys" Oct 31 00:40:27.405942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.405942 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3ea689-f4a7-4b83-afcc-7446124457ae" containerName="revoke-keys" Oct 31 00:40:27.406101 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.406064 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3ea689-f4a7-4b83-afcc-7446124457ae" containerName="revoke-keys" Oct 31 00:40:27.409803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.409779 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.422989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.422963 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:27.423096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.422989 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:40:27.436493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.436442 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw"] Oct 31 00:40:27.592778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.592739 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe0f1be-h5bgw\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.592994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.592859 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe0f1be-h5bgw\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.592994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.592895 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lgvx\" (UniqueName: \"kubernetes.io/projected/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-kube-api-access-9lgvx\") pod \"maas-api-revoke-sub-afe0f1be-h5bgw\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.694509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.694374 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe0f1be-h5bgw\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.694676 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.694535 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe0f1be-h5bgw\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.694676 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.694570 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lgvx\" (UniqueName: \"kubernetes.io/projected/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-kube-api-access-9lgvx\") pod \"maas-api-revoke-sub-afe0f1be-h5bgw\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.695199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.695172 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe0f1be-h5bgw\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.696839 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.696815 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe0f1be-h5bgw\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.717649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.717609 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lgvx\" (UniqueName: \"kubernetes.io/projected/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-kube-api-access-9lgvx\") pod \"maas-api-revoke-sub-afe0f1be-h5bgw\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:27.723820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:27.723788 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:28.021007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:28.020974 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw"] Oct 31 00:40:28.022214 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:40:28.022186 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8bb4b0_f452_43a3_9c1e_26419dbc059f.slice/crio-7bfae20a9b1d767bfff3c367430c9bea34c767bee636df766a4f14c26b8310cb WatchSource:0}: Error finding container 7bfae20a9b1d767bfff3c367430c9bea34c767bee636df766a4f14c26b8310cb: Status 404 returned error can't find the container with id 7bfae20a9b1d767bfff3c367430c9bea34c767bee636df766a4f14c26b8310cb Oct 31 00:40:28.155697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:28.155659 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" event={"ID":"7e8bb4b0-f452-43a3-9c1e-26419dbc059f","Type":"ContainerStarted","Data":"7bfae20a9b1d767bfff3c367430c9bea34c767bee636df766a4f14c26b8310cb"} Oct 31 00:40:28.632283 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:28.632199 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:29.164182 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:29.164103 5978 generic.go:356] "Generic (PLEG): container finished" podID="7e8bb4b0-f452-43a3-9c1e-26419dbc059f" containerID="4007b116ae71f20dd7e4fa63067827726a48408539e05a691bfea2e0b8b1647a" exitCode=0 Oct 31 00:40:29.164571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:29.164183 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" event={"ID":"7e8bb4b0-f452-43a3-9c1e-26419dbc059f","Type":"ContainerDied","Data":"4007b116ae71f20dd7e4fa63067827726a48408539e05a691bfea2e0b8b1647a"} Oct 31 00:40:30.203172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.203077 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:30.333391 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.333351 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:30.532011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.531917 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-kube-api-access-9lgvx\" (UniqueName: \"kubernetes.io/projected/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-kube-api-access-9lgvx\") pod \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " Oct 31 00:40:30.532011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.531968 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-openshift-ca\") pod \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " Oct 31 00:40:30.532382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.532136 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-xks-ca\") pod \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\" (UID: \"7e8bb4b0-f452-43a3-9c1e-26419dbc059f\") " Oct 31 00:40:30.532491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.532415 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-openshift-ca" pod "7e8bb4b0-f452-43a3-9c1e-26419dbc059f" (UID: "7e8bb4b0-f452-43a3-9c1e-26419dbc059f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:30.532848 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.532794 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:30.534333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.534308 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-xks-ca" pod "7e8bb4b0-f452-43a3-9c1e-26419dbc059f" (UID: "7e8bb4b0-f452-43a3-9c1e-26419dbc059f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:30.534409 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.534356 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-kube-api-access-9lgvx" pod "7e8bb4b0-f452-43a3-9c1e-26419dbc059f" (UID: "7e8bb4b0-f452-43a3-9c1e-26419dbc059f"). InnerVolumeSpecName "kube-api-access-9lgvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:30.641185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.641138 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lgvx\" (UniqueName: \"kubernetes.io/projected/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-kube-api-access-9lgvx\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:30.641185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:30.641193 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8bb4b0-f452-43a3-9c1e-26419dbc059f-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:31.177959 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:31.177926 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" Oct 31 00:40:31.178152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:31.177967 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw" event={"ID":"7e8bb4b0-f452-43a3-9c1e-26419dbc059f","Type":"ContainerDied","Data":"7bfae20a9b1d767bfff3c367430c9bea34c767bee636df766a4f14c26b8310cb"} Oct 31 00:40:31.178152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:31.178042 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bfae20a9b1d767bfff3c367430c9bea34c767bee636df766a4f14c26b8310cb" Oct 31 00:40:31.787998 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:31.787951 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:33.182196 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:40:33.182161 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:40:33.497251 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:33.497160 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:35.101013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:35.100980 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:35.988563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:35.988525 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl"] Oct 31 00:40:35.990224 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:35.990188 5978 cpu_manager.go:414] "containerMap: removing container" podUID="7e8bb4b0-f452-43a3-9c1e-26419dbc059f" containerName="revoke-keys" Oct 31 00:40:35.990224 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:35.990221 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8bb4b0-f452-43a3-9c1e-26419dbc059f" containerName="revoke-keys" Oct 31 00:40:35.990566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:35.990535 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8bb4b0-f452-43a3-9c1e-26419dbc059f" containerName="revoke-keys" Oct 31 00:40:35.995634 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:35.995604 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.014134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.014099 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl"] Oct 31 00:40:36.021095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.021066 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:36.021259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.021134 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:40:36.106762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.106721 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccb04086-lj4zl\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.106762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.106758 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccb04086-lj4zl\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.107198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.106933 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ks7\" (UniqueName: \"kubernetes.io/projected/4385d879-e04f-47cf-ac7c-607dc611c9d2-kube-api-access-v2ks7\") pod \"maas-api-revoke-sub-ccb04086-lj4zl\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.208207 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.208177 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccb04086-lj4zl\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.208207 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.208210 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccb04086-lj4zl\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.208485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.208339 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ks7\" (UniqueName: \"kubernetes.io/projected/4385d879-e04f-47cf-ac7c-607dc611c9d2-kube-api-access-v2ks7\") pod \"maas-api-revoke-sub-ccb04086-lj4zl\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.208931 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.208906 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccb04086-lj4zl\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.210726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.210702 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccb04086-lj4zl\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.230203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.230169 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ks7\" (UniqueName: \"kubernetes.io/projected/4385d879-e04f-47cf-ac7c-607dc611c9d2-kube-api-access-v2ks7\") pod \"maas-api-revoke-sub-ccb04086-lj4zl\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.309087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.309001 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:36.610226 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.610189 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl"] Oct 31 00:40:36.611980 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:40:36.611951 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4385d879_e04f_47cf_ac7c_607dc611c9d2.slice/crio-dbfb264852ab340ff25bd6a3607546ea1ffc1ed401927d2ae4817c2b4291478b WatchSource:0}: Error finding container dbfb264852ab340ff25bd6a3607546ea1ffc1ed401927d2ae4817c2b4291478b: Status 404 returned error can't find the container with id dbfb264852ab340ff25bd6a3607546ea1ffc1ed401927d2ae4817c2b4291478b Oct 31 00:40:36.902222 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:36.902127 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:37.241525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:37.241388 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" event={"ID":"4385d879-e04f-47cf-ac7c-607dc611c9d2","Type":"ContainerStarted","Data":"6fa4399e690cf630e89046f7d594c681fe53c97d0c3b41c1b252b118bfffd575"} Oct 31 00:40:37.241525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:37.241519 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" event={"ID":"4385d879-e04f-47cf-ac7c-607dc611c9d2","Type":"ContainerStarted","Data":"dbfb264852ab340ff25bd6a3607546ea1ffc1ed401927d2ae4817c2b4291478b"} Oct 31 00:40:37.282895 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:37.282842 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" podStartSLOduration=2.2828284930000002 podStartE2EDuration="2.282828493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:37.28214039 +0000 UTC m=+1254.980254156" watchObservedRunningTime="2026-10-31 00:40:37.282828493 +0000 UTC m=+1254.980942257" Oct 31 00:40:38.250739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:38.250705 5978 generic.go:356] "Generic (PLEG): container finished" podID="4385d879-e04f-47cf-ac7c-607dc611c9d2" containerID="6fa4399e690cf630e89046f7d594c681fe53c97d0c3b41c1b252b118bfffd575" exitCode=0 Oct 31 00:40:38.251157 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:38.250799 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" event={"ID":"4385d879-e04f-47cf-ac7c-607dc611c9d2","Type":"ContainerDied","Data":"6fa4399e690cf630e89046f7d594c681fe53c97d0c3b41c1b252b118bfffd575"} Oct 31 00:40:38.539431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:38.539333 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:39.468259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.468223 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:39.556480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.556345 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-xks-ca\") pod \"4385d879-e04f-47cf-ac7c-607dc611c9d2\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " Oct 31 00:40:39.556480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.556418 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-openshift-ca\") pod \"4385d879-e04f-47cf-ac7c-607dc611c9d2\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " Oct 31 00:40:39.556719 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.556527 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4385d879-e04f-47cf-ac7c-607dc611c9d2-kube-api-access-v2ks7\" (UniqueName: \"kubernetes.io/projected/4385d879-e04f-47cf-ac7c-607dc611c9d2-kube-api-access-v2ks7\") pod \"4385d879-e04f-47cf-ac7c-607dc611c9d2\" (UID: \"4385d879-e04f-47cf-ac7c-607dc611c9d2\") " Oct 31 00:40:39.556887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.556848 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-openshift-ca" pod "4385d879-e04f-47cf-ac7c-607dc611c9d2" (UID: "4385d879-e04f-47cf-ac7c-607dc611c9d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:39.557152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.557128 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:39.559136 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.559106 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-xks-ca" pod "4385d879-e04f-47cf-ac7c-607dc611c9d2" (UID: "4385d879-e04f-47cf-ac7c-607dc611c9d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:39.559264 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.559174 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4385d879-e04f-47cf-ac7c-607dc611c9d2-kube-api-access-v2ks7" pod "4385d879-e04f-47cf-ac7c-607dc611c9d2" (UID: "4385d879-e04f-47cf-ac7c-607dc611c9d2"). InnerVolumeSpecName "kube-api-access-v2ks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:39.658275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.658238 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ks7\" (UniqueName: \"kubernetes.io/projected/4385d879-e04f-47cf-ac7c-607dc611c9d2-kube-api-access-v2ks7\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:39.658275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:39.658278 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385d879-e04f-47cf-ac7c-607dc611c9d2-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:40.000406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:40.000364 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:40.266094 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:40.266014 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" Oct 31 00:40:40.266094 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:40.266012 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl" event={"ID":"4385d879-e04f-47cf-ac7c-607dc611c9d2","Type":"ContainerDied","Data":"dbfb264852ab340ff25bd6a3607546ea1ffc1ed401927d2ae4817c2b4291478b"} Oct 31 00:40:40.266287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:40.266136 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbfb264852ab340ff25bd6a3607546ea1ffc1ed401927d2ae4817c2b4291478b" Oct 31 00:40:40.290528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:40.290490 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:42.999904 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:42.999861 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:43.143702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:43.143666 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:43.994902 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:43.994868 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:44.500164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.500131 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w"] Oct 31 00:40:44.503942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.503890 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4385d879-e04f-47cf-ac7c-607dc611c9d2" containerName="revoke-keys" Oct 31 00:40:44.503942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.503914 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4385d879-e04f-47cf-ac7c-607dc611c9d2" containerName="revoke-keys" Oct 31 00:40:44.504133 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.504100 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4385d879-e04f-47cf-ac7c-607dc611c9d2" containerName="revoke-keys" Oct 31 00:40:44.548569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.538716 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w"] Oct 31 00:40:44.548569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.538864 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.553013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.552984 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:44.562232 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.562202 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:40:44.704340 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.704297 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771d076e-8td6w\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.704557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.704540 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmc8\" (UniqueName: \"kubernetes.io/projected/bb3d3426-b2ce-49c5-849b-e34598d605e6-kube-api-access-pxmc8\") pod \"maas-api-revoke-sub-771d076e-8td6w\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.704697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.704673 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771d076e-8td6w\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.805695 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.805588 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmc8\" (UniqueName: \"kubernetes.io/projected/bb3d3426-b2ce-49c5-849b-e34598d605e6-kube-api-access-pxmc8\") pod \"maas-api-revoke-sub-771d076e-8td6w\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.805882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.805689 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771d076e-8td6w\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.805882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.805763 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771d076e-8td6w\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.806740 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.806712 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771d076e-8td6w\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.808556 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.808536 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771d076e-8td6w\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.865758 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.863659 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmc8\" (UniqueName: \"kubernetes.io/projected/bb3d3426-b2ce-49c5-849b-e34598d605e6-kube-api-access-pxmc8\") pod \"maas-api-revoke-sub-771d076e-8td6w\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:44.964532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:44.964490 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:45.102519 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:45.102438 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:45.272518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:45.272483 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w"] Oct 31 00:40:45.274049 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:40:45.274015 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3d3426_b2ce_49c5_849b_e34598d605e6.slice/crio-2d72496704dce79a803c8441f92d97b122ba66f470404b2771ebec31a424967a WatchSource:0}: Error finding container 2d72496704dce79a803c8441f92d97b122ba66f470404b2771ebec31a424967a: Status 404 returned error can't find the container with id 2d72496704dce79a803c8441f92d97b122ba66f470404b2771ebec31a424967a Oct 31 00:40:45.317397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:45.317367 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" event={"ID":"bb3d3426-b2ce-49c5-849b-e34598d605e6","Type":"ContainerStarted","Data":"2d72496704dce79a803c8441f92d97b122ba66f470404b2771ebec31a424967a"} Oct 31 00:40:46.327294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:46.327258 5978 generic.go:356] "Generic (PLEG): container finished" podID="bb3d3426-b2ce-49c5-849b-e34598d605e6" containerID="b27c3376fb28fd8adb1738b721e1a9dff9b48bba1491f465b4d0c3b4221ebb15" exitCode=0 Oct 31 00:40:46.327699 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:46.327313 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" event={"ID":"bb3d3426-b2ce-49c5-849b-e34598d605e6","Type":"ContainerDied","Data":"b27c3376fb28fd8adb1738b721e1a9dff9b48bba1491f465b4d0c3b4221ebb15"} Oct 31 00:40:46.787834 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:46.787795 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:46.848346 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:46.848311 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw"] Oct 31 00:40:46.853354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:46.853278 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:46.871347 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:46.871318 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw"] Oct 31 00:40:47.033737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.033695 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0faf990-rfjvw\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.033941 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.033853 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0faf990-rfjvw\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.034015 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.033932 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjw88\" (UniqueName: \"kubernetes.io/projected/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-kube-api-access-hjw88\") pod \"maas-api-revoke-sub-e0faf990-rfjvw\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.117445 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.117403 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv"] Oct 31 00:40:47.121855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.121833 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.134889 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.134864 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0faf990-rfjvw\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.134995 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.134939 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0faf990-rfjvw\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.134995 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.134964 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjw88\" (UniqueName: \"kubernetes.io/projected/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-kube-api-access-hjw88\") pod \"maas-api-revoke-sub-e0faf990-rfjvw\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.135635 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.135611 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0faf990-rfjvw\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.137281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.137255 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0faf990-rfjvw\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.139494 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.139477 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b024b5-openshift-default-dockercfg-gwjft\"" Oct 31 00:40:47.152134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.152103 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv"] Oct 31 00:40:47.164438 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.164405 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjw88\" (UniqueName: \"kubernetes.io/projected/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-kube-api-access-hjw88\") pod \"maas-api-revoke-sub-e0faf990-rfjvw\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.169110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.169083 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:47.236175 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.236138 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.236301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.236194 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.236393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.236378 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.236478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.236409 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.236616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.236592 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.236708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.236653 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.236708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.236682 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snpdk\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-kube-api-access-snpdk\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.236813 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.236722 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.236813 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.236762 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344274 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344321 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344363 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344407 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344432 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpdk\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-kube-api-access-snpdk\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344479 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344507 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344591 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344617 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.344955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.344780 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.345539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.345032 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.345539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.345274 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.345539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.345320 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.345690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.345657 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.347640 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.347612 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.347748 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.347727 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.370667 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.370591 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.370849 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.370826 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpdk\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-kube-api-access-snpdk\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.436308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.436274 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:47.487595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.486132 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw"] Oct 31 00:40:47.492327 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:40:47.492288 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdcf2a5_43ff_431b_9c67_352ebe4f5194.slice/crio-5f94640b66f18adb278ac562dac2eb3835b10cb8f7ea01a8800f05311dc81a1c WatchSource:0}: Error finding container 5f94640b66f18adb278ac562dac2eb3835b10cb8f7ea01a8800f05311dc81a1c: Status 404 returned error can't find the container with id 5f94640b66f18adb278ac562dac2eb3835b10cb8f7ea01a8800f05311dc81a1c Oct 31 00:40:47.495505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.494078 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:47.649805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.649727 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-openshift-ca\") pod \"bb3d3426-b2ce-49c5-849b-e34598d605e6\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " Oct 31 00:40:47.649805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.649780 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-xks-ca\") pod \"bb3d3426-b2ce-49c5-849b-e34598d605e6\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " Oct 31 00:40:47.650054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.650034 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3d3426-b2ce-49c5-849b-e34598d605e6-kube-api-access-pxmc8\" (UniqueName: \"kubernetes.io/projected/bb3d3426-b2ce-49c5-849b-e34598d605e6-kube-api-access-pxmc8\") pod \"bb3d3426-b2ce-49c5-849b-e34598d605e6\" (UID: \"bb3d3426-b2ce-49c5-849b-e34598d605e6\") " Oct 31 00:40:47.650170 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.650148 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-openshift-ca" pod "bb3d3426-b2ce-49c5-849b-e34598d605e6" (UID: "bb3d3426-b2ce-49c5-849b-e34598d605e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:47.650541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.650520 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:47.652259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.652225 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-xks-ca" pod "bb3d3426-b2ce-49c5-849b-e34598d605e6" (UID: "bb3d3426-b2ce-49c5-849b-e34598d605e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:47.652348 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.652253 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3d3426-b2ce-49c5-849b-e34598d605e6-kube-api-access-pxmc8" pod "bb3d3426-b2ce-49c5-849b-e34598d605e6" (UID: "bb3d3426-b2ce-49c5-849b-e34598d605e6"). InnerVolumeSpecName "kube-api-access-pxmc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:47.751654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.751624 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxmc8\" (UniqueName: \"kubernetes.io/projected/bb3d3426-b2ce-49c5-849b-e34598d605e6-kube-api-access-pxmc8\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:47.751654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.751651 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3d3426-b2ce-49c5-849b-e34598d605e6-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:47.761329 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.761305 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv"] Oct 31 00:40:47.762256 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:40:47.762230 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902de6f5_c680_450b_87ed_3c0ad7bd3e88.slice/crio-dbdc104003a879f9d1ceb6aa7002c1cccde2ca3824cb9f7ee74bd5bc20088ca6 WatchSource:0}: Error finding container dbdc104003a879f9d1ceb6aa7002c1cccde2ca3824cb9f7ee74bd5bc20088ca6: Status 404 returned error can't find the container with id dbdc104003a879f9d1ceb6aa7002c1cccde2ca3824cb9f7ee74bd5bc20088ca6 Oct 31 00:40:47.764660 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.764627 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:47.764723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.764694 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:47.764760 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:47.764725 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:40:48.354025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.353976 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" event={"ID":"4bdcf2a5-43ff-431b-9c67-352ebe4f5194","Type":"ContainerStarted","Data":"6425ef992c4154e20ebe349580e75e3b585d2a6f558a57255a2b52e34a51ec20"} Oct 31 00:40:48.354529 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.354044 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" event={"ID":"4bdcf2a5-43ff-431b-9c67-352ebe4f5194","Type":"ContainerStarted","Data":"5f94640b66f18adb278ac562dac2eb3835b10cb8f7ea01a8800f05311dc81a1c"} Oct 31 00:40:48.355474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.355427 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" Oct 31 00:40:48.355572 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.355477 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w" event={"ID":"bb3d3426-b2ce-49c5-849b-e34598d605e6","Type":"ContainerDied","Data":"2d72496704dce79a803c8441f92d97b122ba66f470404b2771ebec31a424967a"} Oct 31 00:40:48.355572 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.355520 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d72496704dce79a803c8441f92d97b122ba66f470404b2771ebec31a424967a" Oct 31 00:40:48.357244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.357210 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" event={"ID":"902de6f5-c680-450b-87ed-3c0ad7bd3e88","Type":"ContainerStarted","Data":"99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127"} Oct 31 00:40:48.357374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.357259 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" event={"ID":"902de6f5-c680-450b-87ed-3c0ad7bd3e88","Type":"ContainerStarted","Data":"dbdc104003a879f9d1ceb6aa7002c1cccde2ca3824cb9f7ee74bd5bc20088ca6"} Oct 31 00:40:48.392548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.392488 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" podStartSLOduration=2.392471198 podStartE2EDuration="2.392471198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:48.38981606 +0000 UTC m=+1266.087929824" watchObservedRunningTime="2026-10-31 00:40:48.392471198 +0000 UTC m=+1266.090584971" Oct 31 00:40:48.436982 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.436912 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" podStartSLOduration=1.436893403 podStartE2EDuration="1.436893403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:48.436026346 +0000 UTC m=+1266.134140124" watchObservedRunningTime="2026-10-31 00:40:48.436893403 +0000 UTC m=+1266.135007168" Oct 31 00:40:48.441656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.441613 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:40:48.441851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.441674 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:40:48.443376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.443354 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:48.690299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:48.690262 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:49.365068 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:49.365028 5978 generic.go:356] "Generic (PLEG): container finished" podID="4bdcf2a5-43ff-431b-9c67-352ebe4f5194" containerID="6425ef992c4154e20ebe349580e75e3b585d2a6f558a57255a2b52e34a51ec20" exitCode=0 Oct 31 00:40:49.365596 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:49.365115 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" event={"ID":"4bdcf2a5-43ff-431b-9c67-352ebe4f5194","Type":"ContainerDied","Data":"6425ef992c4154e20ebe349580e75e3b585d2a6f558a57255a2b52e34a51ec20"} Oct 31 00:40:49.436908 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:49.436866 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:40:49.437062 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:49.436929 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:40:50.439998 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.439943 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:40:50.440552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.440030 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:40:50.540678 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.540639 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:50.592527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.592499 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:50.684541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.684423 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-kube-api-access-hjw88\" (UniqueName: \"kubernetes.io/projected/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-kube-api-access-hjw88\") pod \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " Oct 31 00:40:50.684541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.684521 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-openshift-ca\") pod \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " Oct 31 00:40:50.684764 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.684578 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-xks-ca\") pod \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\" (UID: \"4bdcf2a5-43ff-431b-9c67-352ebe4f5194\") " Oct 31 00:40:50.684875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.684850 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-openshift-ca" pod "4bdcf2a5-43ff-431b-9c67-352ebe4f5194" (UID: "4bdcf2a5-43ff-431b-9c67-352ebe4f5194"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:50.685050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.685037 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:50.686607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.686584 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-xks-ca" pod "4bdcf2a5-43ff-431b-9c67-352ebe4f5194" (UID: "4bdcf2a5-43ff-431b-9c67-352ebe4f5194"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:50.686722 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.686654 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-kube-api-access-hjw88" pod "4bdcf2a5-43ff-431b-9c67-352ebe4f5194" (UID: "4bdcf2a5-43ff-431b-9c67-352ebe4f5194"). InnerVolumeSpecName "kube-api-access-hjw88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:50.786321 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.786286 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:50.786321 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:50.786318 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjw88\" (UniqueName: \"kubernetes.io/projected/4bdcf2a5-43ff-431b-9c67-352ebe4f5194-kube-api-access-hjw88\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:40:51.379624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:51.379577 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" event={"ID":"4bdcf2a5-43ff-431b-9c67-352ebe4f5194","Type":"ContainerDied","Data":"5f94640b66f18adb278ac562dac2eb3835b10cb8f7ea01a8800f05311dc81a1c"} Oct 31 00:40:51.380116 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:51.379664 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f94640b66f18adb278ac562dac2eb3835b10cb8f7ea01a8800f05311dc81a1c" Oct 31 00:40:51.380116 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:51.379587 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw" Oct 31 00:40:51.438001 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:51.437950 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:40:51.438196 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:51.438023 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:40:52.230643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:52.230606 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:52.441636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:52.441600 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:40:52.441864 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:52.441650 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:40:53.436921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:53.436878 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:40:53.437328 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:53.436943 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:40:53.921294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:53.921255 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:54.476167 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:54.476127 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:54.477209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:54.477185 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:54.478323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:54.478304 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 00:40:55.527937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:55.527903 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:57.212642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.212604 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:57.412784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.412735 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg"] Oct 31 00:40:57.413367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.413349 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4bdcf2a5-43ff-431b-9c67-352ebe4f5194" containerName="revoke-keys" Oct 31 00:40:57.413367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.413366 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bdcf2a5-43ff-431b-9c67-352ebe4f5194" containerName="revoke-keys" Oct 31 00:40:57.413592 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.413392 5978 cpu_manager.go:414] "containerMap: removing container" podUID="bb3d3426-b2ce-49c5-849b-e34598d605e6" containerName="revoke-keys" Oct 31 00:40:57.413592 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.413397 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3d3426-b2ce-49c5-849b-e34598d605e6" containerName="revoke-keys" Oct 31 00:40:57.413592 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.413507 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bdcf2a5-43ff-431b-9c67-352ebe4f5194" containerName="revoke-keys" Oct 31 00:40:57.413592 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.413521 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3d3426-b2ce-49c5-849b-e34598d605e6" containerName="revoke-keys" Oct 31 00:40:57.417942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.417904 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.473996 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.473914 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:57.478560 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.478531 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg"] Oct 31 00:40:57.478827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.478809 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:40:57.489756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.489726 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ba702f-7vvzg\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.489938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.489916 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9kwn\" (UniqueName: \"kubernetes.io/projected/875a4139-a4c8-4eae-81fb-961a0321257c-kube-api-access-p9kwn\") pod \"maas-api-revoke-sub-31ba702f-7vvzg\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.490011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.489989 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ba702f-7vvzg\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.529860 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.529831 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:57.590954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.590915 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ba702f-7vvzg\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.591142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.591010 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9kwn\" (UniqueName: \"kubernetes.io/projected/875a4139-a4c8-4eae-81fb-961a0321257c-kube-api-access-p9kwn\") pod \"maas-api-revoke-sub-31ba702f-7vvzg\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.591142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.591049 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ba702f-7vvzg\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.591830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.591798 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ba702f-7vvzg\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.593575 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.593550 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ba702f-7vvzg\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.613679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.613644 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9kwn\" (UniqueName: \"kubernetes.io/projected/875a4139-a4c8-4eae-81fb-961a0321257c-kube-api-access-p9kwn\") pod \"maas-api-revoke-sub-31ba702f-7vvzg\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:57.733021 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:57.732943 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:40:58.049901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:58.049643 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg"] Oct 31 00:40:58.066434 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:40:58.061090 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875a4139_a4c8_4eae_81fb_961a0321257c.slice/crio-2d619f687dc0638bdad6a2a70a74cd27123b86ed861bf84f814ad803a6abae5c WatchSource:0}: Error finding container 2d619f687dc0638bdad6a2a70a74cd27123b86ed861bf84f814ad803a6abae5c: Status 404 returned error can't find the container with id 2d619f687dc0638bdad6a2a70a74cd27123b86ed861bf84f814ad803a6abae5c Oct 31 00:40:58.706000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:58.705894 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" event={"ID":"875a4139-a4c8-4eae-81fb-961a0321257c","Type":"ContainerStarted","Data":"2d619f687dc0638bdad6a2a70a74cd27123b86ed861bf84f814ad803a6abae5c"} Oct 31 00:40:59.750551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:59.748848 5978 generic.go:356] "Generic (PLEG): container finished" podID="875a4139-a4c8-4eae-81fb-961a0321257c" containerID="c28cf00cd168056f23226b01c9658d88af87cbf61d7a95afc9cff1b87e32d054" exitCode=0 Oct 31 00:40:59.750551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:40:59.748975 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" event={"ID":"875a4139-a4c8-4eae-81fb-961a0321257c","Type":"ContainerDied","Data":"c28cf00cd168056f23226b01c9658d88af87cbf61d7a95afc9cff1b87e32d054"} Oct 31 00:41:00.631356 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:00.631320 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:00.981146 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:00.981116 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:41:01.115209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.114363 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-openshift-ca\") pod \"875a4139-a4c8-4eae-81fb-961a0321257c\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " Oct 31 00:41:01.115209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.114605 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875a4139-a4c8-4eae-81fb-961a0321257c-kube-api-access-p9kwn\" (UniqueName: \"kubernetes.io/projected/875a4139-a4c8-4eae-81fb-961a0321257c-kube-api-access-p9kwn\") pod \"875a4139-a4c8-4eae-81fb-961a0321257c\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " Oct 31 00:41:01.115209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.114662 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-xks-ca\") pod \"875a4139-a4c8-4eae-81fb-961a0321257c\" (UID: \"875a4139-a4c8-4eae-81fb-961a0321257c\") " Oct 31 00:41:01.115209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.114839 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-openshift-ca" pod "875a4139-a4c8-4eae-81fb-961a0321257c" (UID: "875a4139-a4c8-4eae-81fb-961a0321257c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:01.115209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.115172 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:01.118797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.118749 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-xks-ca" pod "875a4139-a4c8-4eae-81fb-961a0321257c" (UID: "875a4139-a4c8-4eae-81fb-961a0321257c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:01.119123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.119082 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875a4139-a4c8-4eae-81fb-961a0321257c-kube-api-access-p9kwn" pod "875a4139-a4c8-4eae-81fb-961a0321257c" (UID: "875a4139-a4c8-4eae-81fb-961a0321257c"). InnerVolumeSpecName "kube-api-access-p9kwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:01.217011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.216884 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9kwn\" (UniqueName: \"kubernetes.io/projected/875a4139-a4c8-4eae-81fb-961a0321257c-kube-api-access-p9kwn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:01.217011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.216922 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875a4139-a4c8-4eae-81fb-961a0321257c-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:01.774081 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.774042 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" Oct 31 00:41:01.774274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.774085 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg" event={"ID":"875a4139-a4c8-4eae-81fb-961a0321257c","Type":"ContainerDied","Data":"2d619f687dc0638bdad6a2a70a74cd27123b86ed861bf84f814ad803a6abae5c"} Oct 31 00:41:01.774274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:01.774148 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d619f687dc0638bdad6a2a70a74cd27123b86ed861bf84f814ad803a6abae5c" Oct 31 00:41:02.328162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.328125 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9"] Oct 31 00:41:02.328816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.328800 5978 cpu_manager.go:414] "containerMap: removing container" podUID="875a4139-a4c8-4eae-81fb-961a0321257c" containerName="revoke-keys" Oct 31 00:41:02.328866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.328818 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875a4139-a4c8-4eae-81fb-961a0321257c" containerName="revoke-keys" Oct 31 00:41:02.328923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.328914 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="875a4139-a4c8-4eae-81fb-961a0321257c" containerName="revoke-keys" Oct 31 00:41:02.332575 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.332554 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.345036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.345012 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:41:02.354735 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.354703 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:41:02.357992 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.357960 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9"] Oct 31 00:41:02.433559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.433518 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9461f01-nnmf9\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.433743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.433575 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9461f01-nnmf9\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.433786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.433737 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmqks\" (UniqueName: \"kubernetes.io/projected/8867d6c0-ae02-4c82-9aed-bd517ed149d0-kube-api-access-cmqks\") pod \"maas-api-revoke-sub-d9461f01-nnmf9\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.503056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.503014 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:02.535066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.535025 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9461f01-nnmf9\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.535329 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.535077 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9461f01-nnmf9\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.535417 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.535352 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmqks\" (UniqueName: \"kubernetes.io/projected/8867d6c0-ae02-4c82-9aed-bd517ed149d0-kube-api-access-cmqks\") pod \"maas-api-revoke-sub-d9461f01-nnmf9\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.535905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.535876 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9461f01-nnmf9\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.537547 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.537526 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9461f01-nnmf9\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.559932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.559896 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmqks\" (UniqueName: \"kubernetes.io/projected/8867d6c0-ae02-4c82-9aed-bd517ed149d0-kube-api-access-cmqks\") pod \"maas-api-revoke-sub-d9461f01-nnmf9\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.646994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.646903 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:02.953469 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:02.953426 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9"] Oct 31 00:41:02.954640 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:02.954612 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8867d6c0_ae02_4c82_9aed_bd517ed149d0.slice/crio-0d21239094cd778050c2f9ccf0748e775bfb86ac73ce7d687c54b1a77a8ce511 WatchSource:0}: Error finding container 0d21239094cd778050c2f9ccf0748e775bfb86ac73ce7d687c54b1a77a8ce511: Status 404 returned error can't find the container with id 0d21239094cd778050c2f9ccf0748e775bfb86ac73ce7d687c54b1a77a8ce511 Oct 31 00:41:03.182844 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:41:03.182807 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:41:03.796218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:03.796181 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" event={"ID":"8867d6c0-ae02-4c82-9aed-bd517ed149d0","Type":"ContainerStarted","Data":"168a5a52a8f252d9e1c328fc8789653d5ac9735d6c236ab29218aa8536ae299a"} Oct 31 00:41:03.796657 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:03.796236 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" event={"ID":"8867d6c0-ae02-4c82-9aed-bd517ed149d0","Type":"ContainerStarted","Data":"0d21239094cd778050c2f9ccf0748e775bfb86ac73ce7d687c54b1a77a8ce511"} Oct 31 00:41:03.834793 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:03.834739 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" podStartSLOduration=1.8347254610000001 podStartE2EDuration="1.834725461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:03.831946035 +0000 UTC m=+1281.530059815" watchObservedRunningTime="2026-10-31 00:41:03.834725461 +0000 UTC m=+1281.532839256" Oct 31 00:41:04.192506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:04.192471 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:04.805173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:04.805141 5978 generic.go:356] "Generic (PLEG): container finished" podID="8867d6c0-ae02-4c82-9aed-bd517ed149d0" containerID="168a5a52a8f252d9e1c328fc8789653d5ac9735d6c236ab29218aa8536ae299a" exitCode=0 Oct 31 00:41:04.805593 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:04.805227 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" event={"ID":"8867d6c0-ae02-4c82-9aed-bd517ed149d0","Type":"ContainerDied","Data":"168a5a52a8f252d9e1c328fc8789653d5ac9735d6c236ab29218aa8536ae299a"} Oct 31 00:41:05.830510 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:05.830465 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:05.970767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:05.970740 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:06.093137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.093105 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-xks-ca\") pod \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " Oct 31 00:41:06.093291 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.093227 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8867d6c0-ae02-4c82-9aed-bd517ed149d0-kube-api-access-cmqks\" (UniqueName: \"kubernetes.io/projected/8867d6c0-ae02-4c82-9aed-bd517ed149d0-kube-api-access-cmqks\") pod \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " Oct 31 00:41:06.093353 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.093289 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-openshift-ca\") pod \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\" (UID: \"8867d6c0-ae02-4c82-9aed-bd517ed149d0\") " Oct 31 00:41:06.093688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.093655 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-openshift-ca" pod "8867d6c0-ae02-4c82-9aed-bd517ed149d0" (UID: "8867d6c0-ae02-4c82-9aed-bd517ed149d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:06.095256 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.095233 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-xks-ca" pod "8867d6c0-ae02-4c82-9aed-bd517ed149d0" (UID: "8867d6c0-ae02-4c82-9aed-bd517ed149d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:06.095363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.095263 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8867d6c0-ae02-4c82-9aed-bd517ed149d0-kube-api-access-cmqks" pod "8867d6c0-ae02-4c82-9aed-bd517ed149d0" (UID: "8867d6c0-ae02-4c82-9aed-bd517ed149d0"). InnerVolumeSpecName "kube-api-access-cmqks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:06.194746 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.194710 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:06.194746 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.194742 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmqks\" (UniqueName: \"kubernetes.io/projected/8867d6c0-ae02-4c82-9aed-bd517ed149d0-kube-api-access-cmqks\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:06.194746 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.194756 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8867d6c0-ae02-4c82-9aed-bd517ed149d0-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:06.253533 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.253492 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp"] Oct 31 00:41:06.254768 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.254731 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8867d6c0-ae02-4c82-9aed-bd517ed149d0" containerName="revoke-keys" Oct 31 00:41:06.254768 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.254763 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8867d6c0-ae02-4c82-9aed-bd517ed149d0" containerName="revoke-keys" Oct 31 00:41:06.254949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.254895 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8867d6c0-ae02-4c82-9aed-bd517ed149d0" containerName="revoke-keys" Oct 31 00:41:06.259142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.259117 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.272843 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.272807 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"openshift-service-ca.crt\"" Oct 31 00:41:06.281907 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.281881 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"default-dockercfg-cbmxc\"" Oct 31 00:41:06.282148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.282124 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"distinct-w1-b024b5-kserve-self-signed-certs\"" Oct 31 00:41:06.282233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.282167 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"kube-root-ca.crt\"" Oct 31 00:41:06.285311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.285284 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp"] Oct 31 00:41:06.347885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.347797 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:06.397775 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.397738 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4bab9-775a-4571-b7aa-2ec73409539f-tls-certs\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.398006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.397791 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-dshm\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.398006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.397912 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khxfm\" (UniqueName: \"kubernetes.io/projected/dcf4bab9-775a-4571-b7aa-2ec73409539f-kube-api-access-khxfm\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.398006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.397951 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-model-cache\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.398164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.398073 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-home\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.398164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.398096 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-tmp-dir\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.500674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.500077 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4bab9-775a-4571-b7aa-2ec73409539f-tls-certs\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.500674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.500166 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-dshm\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.500674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.500258 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khxfm\" (UniqueName: \"kubernetes.io/projected/dcf4bab9-775a-4571-b7aa-2ec73409539f-kube-api-access-khxfm\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.500674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.500285 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-model-cache\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.500674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.500399 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-home\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.500674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.500432 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-tmp-dir\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.501169 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.500980 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-home\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.501169 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.501060 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-tmp-dir\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.501392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.501371 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-model-cache\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.503743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.503717 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-dshm\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.511320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.511281 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4bab9-775a-4571-b7aa-2ec73409539f-tls-certs\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.523330 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.523301 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khxfm\" (UniqueName: \"kubernetes.io/projected/dcf4bab9-775a-4571-b7aa-2ec73409539f-kube-api-access-khxfm\") pod \"distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.576322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.576283 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:06.821547 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.821519 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" Oct 31 00:41:06.821704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.821551 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9" event={"ID":"8867d6c0-ae02-4c82-9aed-bd517ed149d0","Type":"ContainerDied","Data":"0d21239094cd778050c2f9ccf0748e775bfb86ac73ce7d687c54b1a77a8ce511"} Oct 31 00:41:06.821704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.821608 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d21239094cd778050c2f9ccf0748e775bfb86ac73ce7d687c54b1a77a8ce511" Oct 31 00:41:06.884091 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:06.884054 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp"] Oct 31 00:41:06.885226 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:06.885198 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcf4bab9_775a_4571_b7aa_2ec73409539f.slice/crio-0a184e65cfea33939510816100cc99bc07ed0d3cad39820b9c0439377937dac3 WatchSource:0}: Error finding container 0a184e65cfea33939510816100cc99bc07ed0d3cad39820b9c0439377937dac3: Status 404 returned error can't find the container with id 0a184e65cfea33939510816100cc99bc07ed0d3cad39820b9c0439377937dac3 Oct 31 00:41:07.531084 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:07.531043 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:07.829926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:07.829886 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" event={"ID":"dcf4bab9-775a-4571-b7aa-2ec73409539f","Type":"ContainerStarted","Data":"1fa2cb88f3a4c74eeb2fc83e47dd58964ad1e3b73e3fdda862c965088b31c7b1"} Oct 31 00:41:07.830126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:07.829935 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" event={"ID":"dcf4bab9-775a-4571-b7aa-2ec73409539f","Type":"ContainerStarted","Data":"0a184e65cfea33939510816100cc99bc07ed0d3cad39820b9c0439377937dac3"} Oct 31 00:41:07.867220 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:07.867170 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podStartSLOduration=1.8671568920000001 podStartE2EDuration="1.867156892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:07.866793588 +0000 UTC m=+1285.564907353" watchObservedRunningTime="2026-10-31 00:41:07.867156892 +0000 UTC m=+1285.565270657" Oct 31 00:41:08.606476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:08.595544 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:10.198699 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:10.198661 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:11.697024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.696982 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp"] Oct 31 00:41:11.701598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.701560 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:11.701746 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.701661 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:11.714693 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.714669 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:41:11.724812 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.724780 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:41:11.729134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.729003 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp"] Oct 31 00:41:11.873144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.873107 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-995e74cb-2nrrp\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:11.873327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.873227 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dklg5\" (UniqueName: \"kubernetes.io/projected/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-kube-api-access-dklg5\") pod \"maas-api-revoke-sub-995e74cb-2nrrp\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:11.873368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.873331 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-995e74cb-2nrrp\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:11.965187 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.965096 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:11.974431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.974397 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklg5\" (UniqueName: \"kubernetes.io/projected/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-kube-api-access-dklg5\") pod \"maas-api-revoke-sub-995e74cb-2nrrp\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:11.974673 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.974499 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-995e74cb-2nrrp\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:11.974673 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.974608 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-995e74cb-2nrrp\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:11.975417 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.975391 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-995e74cb-2nrrp\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:11.976971 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.976945 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-995e74cb-2nrrp\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:11.995925 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:11.995895 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklg5\" (UniqueName: \"kubernetes.io/projected/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-kube-api-access-dklg5\") pod \"maas-api-revoke-sub-995e74cb-2nrrp\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:12.016916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:12.016884 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:12.323493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:12.318363 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp"] Oct 31 00:41:12.323493 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:12.318797 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9eec2f_ea9e_4edb_bc72_cbca1ac91eae.slice/crio-bed1a2055b6b80765b4f9144669ecc39cd4081f2681a985c8faad5f58655bedd WatchSource:0}: Error finding container bed1a2055b6b80765b4f9144669ecc39cd4081f2681a985c8faad5f58655bedd: Status 404 returned error can't find the container with id bed1a2055b6b80765b4f9144669ecc39cd4081f2681a985c8faad5f58655bedd Oct 31 00:41:12.869716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:12.869676 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" event={"ID":"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae","Type":"ContainerStarted","Data":"6682e18f55acd51fd8f9c2975bed4acf52a885b6a61d02e9655b2cf910229e85"} Oct 31 00:41:12.870631 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:12.869743 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" event={"ID":"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae","Type":"ContainerStarted","Data":"bed1a2055b6b80765b4f9144669ecc39cd4081f2681a985c8faad5f58655bedd"} Oct 31 00:41:12.910109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:12.910005 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" podStartSLOduration=1.9099897700000001 podStartE2EDuration="1.90998977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:12.906974672 +0000 UTC m=+1290.605088434" watchObservedRunningTime="2026-10-31 00:41:12.90998977 +0000 UTC m=+1290.608103534" Oct 31 00:41:13.624445 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.624394 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq"] Oct 31 00:41:13.629916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.629887 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.653579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.653550 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-671959\"/\"distinct-w0-671959-kserve-self-signed-certs\"" Oct 31 00:41:13.657256 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.657224 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq"] Oct 31 00:41:13.688344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.688305 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:13.798963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.798928 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee1ccd5-48b3-48db-839b-b85f9d78997b-tls-certs\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.799123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.799011 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-dshm\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.799123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.799104 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbt9h\" (UniqueName: \"kubernetes.io/projected/aee1ccd5-48b3-48db-839b-b85f9d78997b-kube-api-access-dbt9h\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.799205 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.799166 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-model-cache\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.799275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.799222 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-tmp-dir\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.799275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.799249 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-home\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.837236 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.837196 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:13.881770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.881663 5978 generic.go:356] "Generic (PLEG): container finished" podID="3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae" containerID="6682e18f55acd51fd8f9c2975bed4acf52a885b6a61d02e9655b2cf910229e85" exitCode=0 Oct 31 00:41:13.881770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.881729 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" event={"ID":"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae","Type":"ContainerDied","Data":"6682e18f55acd51fd8f9c2975bed4acf52a885b6a61d02e9655b2cf910229e85"} Oct 31 00:41:13.900097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.900056 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-tmp-dir\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.900097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.900097 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-home\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.900368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.900215 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee1ccd5-48b3-48db-839b-b85f9d78997b-tls-certs\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.900368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.900245 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-dshm\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.900368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.900349 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbt9h\" (UniqueName: \"kubernetes.io/projected/aee1ccd5-48b3-48db-839b-b85f9d78997b-kube-api-access-dbt9h\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.900564 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.900380 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-model-cache\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.900564 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.900469 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-tmp-dir\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.900564 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.900540 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-home\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.900722 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.900674 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-model-cache\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.902720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.902694 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-dshm\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.902832 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.902818 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee1ccd5-48b3-48db-839b-b85f9d78997b-tls-certs\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.926126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.926096 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbt9h\" (UniqueName: \"kubernetes.io/projected/aee1ccd5-48b3-48db-839b-b85f9d78997b-kube-api-access-dbt9h\") pod \"distinct-w0-671959-kserve-85df5595f4-hbddq\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:13.945046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:13.945012 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:14.245597 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:14.245564 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq"] Oct 31 00:41:14.248188 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:14.248159 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaee1ccd5_48b3_48db_839b_b85f9d78997b.slice/crio-e67ca95a8e5cff31f404b87f6177f69f7c244fe56d7e5d2b6941e0e4ed00d7a9 WatchSource:0}: Error finding container e67ca95a8e5cff31f404b87f6177f69f7c244fe56d7e5d2b6941e0e4ed00d7a9: Status 404 returned error can't find the container with id e67ca95a8e5cff31f404b87f6177f69f7c244fe56d7e5d2b6941e0e4ed00d7a9 Oct 31 00:41:14.890390 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:14.890356 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" event={"ID":"aee1ccd5-48b3-48db-839b-b85f9d78997b","Type":"ContainerStarted","Data":"45efc917250b6eed75f20c6944673a620537f0729191a2455f2eafe6233823ca"} Oct 31 00:41:14.890390 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:14.890400 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" event={"ID":"aee1ccd5-48b3-48db-839b-b85f9d78997b","Type":"ContainerStarted","Data":"e67ca95a8e5cff31f404b87f6177f69f7c244fe56d7e5d2b6941e0e4ed00d7a9"} Oct 31 00:41:14.929441 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:14.929379 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podStartSLOduration=1.9293625460000001 podStartE2EDuration="1.929362546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:14.927234455 +0000 UTC m=+1292.625348220" watchObservedRunningTime="2026-10-31 00:41:14.929362546 +0000 UTC m=+1292.627476581" Oct 31 00:41:14.945381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:14.945346 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:15.037875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.037850 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:15.112611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.112576 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-xks-ca\") pod \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " Oct 31 00:41:15.112942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.112916 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-openshift-ca\") pod \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " Oct 31 00:41:15.113068 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.113052 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-kube-api-access-dklg5\" (UniqueName: \"kubernetes.io/projected/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-kube-api-access-dklg5\") pod \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\" (UID: \"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae\") " Oct 31 00:41:15.113317 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.113283 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-openshift-ca" pod "3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae" (UID: "3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:15.113701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.113677 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:15.115127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.115102 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-xks-ca" pod "3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae" (UID: "3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:15.115673 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.115643 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-kube-api-access-dklg5" pod "3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae" (UID: "3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae"). InnerVolumeSpecName "kube-api-access-dklg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:15.144446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.144352 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp"] Oct 31 00:41:15.145327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.145300 5978 cpu_manager.go:414] "containerMap: removing container" podUID="3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae" containerName="revoke-keys" Oct 31 00:41:15.145447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.145327 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae" containerName="revoke-keys" Oct 31 00:41:15.145544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.145536 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae" containerName="revoke-keys" Oct 31 00:41:15.149910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.149886 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.167859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.167500 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp"] Oct 31 00:41:15.215089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.215056 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac57320-x8trp\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.215245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.215164 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2pk2\" (UniqueName: \"kubernetes.io/projected/50c75715-9366-48ae-8d71-c1ea2cdc0484-kube-api-access-g2pk2\") pod \"maas-api-revoke-sub-aac57320-x8trp\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.215245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.215221 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac57320-x8trp\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.215345 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.215332 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:15.215345 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.215344 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dklg5\" (UniqueName: \"kubernetes.io/projected/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae-kube-api-access-dklg5\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:15.316834 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.316800 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac57320-x8trp\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.317010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.316880 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2pk2\" (UniqueName: \"kubernetes.io/projected/50c75715-9366-48ae-8d71-c1ea2cdc0484-kube-api-access-g2pk2\") pod \"maas-api-revoke-sub-aac57320-x8trp\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.317010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.316939 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac57320-x8trp\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.317658 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.317630 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac57320-x8trp\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.319194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.319172 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac57320-x8trp\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.338883 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.338850 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2pk2\" (UniqueName: \"kubernetes.io/projected/50c75715-9366-48ae-8d71-c1ea2cdc0484-kube-api-access-g2pk2\") pod \"maas-api-revoke-sub-aac57320-x8trp\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.466552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.466445 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:15.514035 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.513999 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:15.766043 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.766007 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp"] Oct 31 00:41:15.766900 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:15.766874 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c75715_9366_48ae_8d71_c1ea2cdc0484.slice/crio-92b9769430419d5ed48717205baefd92a3ddd0bf0aa761a286e99b595bcbb05f WatchSource:0}: Error finding container 92b9769430419d5ed48717205baefd92a3ddd0bf0aa761a286e99b595bcbb05f: Status 404 returned error can't find the container with id 92b9769430419d5ed48717205baefd92a3ddd0bf0aa761a286e99b595bcbb05f Oct 31 00:41:15.898784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.898754 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" Oct 31 00:41:15.899242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.898754 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp" event={"ID":"3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae","Type":"ContainerDied","Data":"bed1a2055b6b80765b4f9144669ecc39cd4081f2681a985c8faad5f58655bedd"} Oct 31 00:41:15.899242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.898902 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed1a2055b6b80765b4f9144669ecc39cd4081f2681a985c8faad5f58655bedd" Oct 31 00:41:15.900269 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:15.900241 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" event={"ID":"50c75715-9366-48ae-8d71-c1ea2cdc0484","Type":"ContainerStarted","Data":"92b9769430419d5ed48717205baefd92a3ddd0bf0aa761a286e99b595bcbb05f"} Oct 31 00:41:16.577193 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:16.577152 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:16.589988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:16.589956 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:16.908262 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:16.908172 5978 generic.go:356] "Generic (PLEG): container finished" podID="50c75715-9366-48ae-8d71-c1ea2cdc0484" containerID="23ecca57b075ffbba777c914e5d6de9c554760eff08520014c706c71f57c59da" exitCode=0 Oct 31 00:41:16.908828 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:16.908256 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" event={"ID":"50c75715-9366-48ae-8d71-c1ea2cdc0484","Type":"ContainerDied","Data":"23ecca57b075ffbba777c914e5d6de9c554760eff08520014c706c71f57c59da"} Oct 31 00:41:16.919927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:16.919899 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:41:17.332055 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.332010 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:17.483564 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.483535 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s"] Oct 31 00:41:17.487948 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.487924 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.514487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.514426 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s"] Oct 31 00:41:17.546417 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.546366 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.546644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.546446 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.546644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.546532 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7474b67-7515-46f1-9b80-09c2353018a0-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.546775 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.546713 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.546775 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.546757 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.546955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.546937 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.547028 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.546983 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.547075 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.547053 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7474b67-7515-46f1-9b80-09c2353018a0-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.547121 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.547080 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc82d\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-kube-api-access-gc82d\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.647841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.647756 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.647841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.647793 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7474b67-7515-46f1-9b80-09c2353018a0-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.647841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.647808 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc82d\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-kube-api-access-gc82d\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.647864 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.647920 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.647938 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7474b67-7515-46f1-9b80-09c2353018a0-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.648013 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.648042 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.648163 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.648244 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.648303 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.648479 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648590 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.648573 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.648761 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.648740 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7474b67-7515-46f1-9b80-09c2353018a0-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.650226 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.650208 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.650650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.650633 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7474b67-7515-46f1-9b80-09c2353018a0-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.668999 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.668971 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc82d\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-kube-api-access-gc82d\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.669383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.669365 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-xw55s\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.724978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.724942 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb"] Oct 31 00:41:17.730436 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.730410 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.749112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.749071 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcp22\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-kube-api-access-zcp22\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.749112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.749117 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.749355 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.749326 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.749435 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.749372 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.749538 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.749487 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.749538 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.749515 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.749607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.749584 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091d8d89-5935-4a95-a9c4-882e48110003-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.749638 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.749621 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.749672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.749645 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091d8d89-5935-4a95-a9c4-882e48110003-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.752417 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.752391 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb"] Oct 31 00:41:17.804223 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.804184 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:17.850882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.850847 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.850882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.850886 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.851142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.850920 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.851142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.850938 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.851142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.850976 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091d8d89-5935-4a95-a9c4-882e48110003-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.851142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.850997 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.851142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.851019 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091d8d89-5935-4a95-a9c4-882e48110003-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.851142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.851107 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcp22\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-kube-api-access-zcp22\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.851142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.851127 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.851560 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.851534 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.852113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.851969 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.852624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.851988 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.853042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.853018 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.853548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.853527 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091d8d89-5935-4a95-a9c4-882e48110003-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.854526 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.854500 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.855214 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.855191 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091d8d89-5935-4a95-a9c4-882e48110003-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.874204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.874167 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcp22\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-kube-api-access-zcp22\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:17.877310 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:17.877257 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:18.045281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.045217 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:18.052766 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.052745 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:18.122729 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.122696 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s"] Oct 31 00:41:18.123667 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:18.123636 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7474b67_7515_46f1_9b80_09c2353018a0.slice/crio-7d5782cdcb635556105d90b13ca54ccbc285911e52e7d623dd85901278152bbb WatchSource:0}: Error finding container 7d5782cdcb635556105d90b13ca54ccbc285911e52e7d623dd85901278152bbb: Status 404 returned error can't find the container with id 7d5782cdcb635556105d90b13ca54ccbc285911e52e7d623dd85901278152bbb Oct 31 00:41:18.126419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.126382 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:18.126572 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.126550 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:18.126661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.126606 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:18.153951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.153921 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c75715-9366-48ae-8d71-c1ea2cdc0484-kube-api-access-g2pk2\" (UniqueName: \"kubernetes.io/projected/50c75715-9366-48ae-8d71-c1ea2cdc0484-kube-api-access-g2pk2\") pod \"50c75715-9366-48ae-8d71-c1ea2cdc0484\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " Oct 31 00:41:18.154110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.154002 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-openshift-ca\") pod \"50c75715-9366-48ae-8d71-c1ea2cdc0484\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " Oct 31 00:41:18.154314 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.154293 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-xks-ca\") pod \"50c75715-9366-48ae-8d71-c1ea2cdc0484\" (UID: \"50c75715-9366-48ae-8d71-c1ea2cdc0484\") " Oct 31 00:41:18.154479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.154439 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-openshift-ca" pod "50c75715-9366-48ae-8d71-c1ea2cdc0484" (UID: "50c75715-9366-48ae-8d71-c1ea2cdc0484"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:18.154929 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.154906 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:18.156208 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.156182 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c75715-9366-48ae-8d71-c1ea2cdc0484-kube-api-access-g2pk2" pod "50c75715-9366-48ae-8d71-c1ea2cdc0484" (UID: "50c75715-9366-48ae-8d71-c1ea2cdc0484"). InnerVolumeSpecName "kube-api-access-g2pk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:18.156329 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.156304 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-xks-ca" pod "50c75715-9366-48ae-8d71-c1ea2cdc0484" (UID: "50c75715-9366-48ae-8d71-c1ea2cdc0484"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:18.233460 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.233404 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh"] Oct 31 00:41:18.234143 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.234122 5978 cpu_manager.go:414] "containerMap: removing container" podUID="50c75715-9366-48ae-8d71-c1ea2cdc0484" containerName="revoke-keys" Oct 31 00:41:18.234250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.234144 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c75715-9366-48ae-8d71-c1ea2cdc0484" containerName="revoke-keys" Oct 31 00:41:18.234325 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.234308 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c75715-9366-48ae-8d71-c1ea2cdc0484" containerName="revoke-keys" Oct 31 00:41:18.238060 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.238039 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.255900 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.255849 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h9l8\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-kube-api-access-6h9l8\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.256074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.255924 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.256074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.255992 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.256074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.256043 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.256245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.256144 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.256245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.256185 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.256245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.256214 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d476266c-048c-469f-9fc6-7bad2061ea9c-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.256245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.256241 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.256486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.256272 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.256486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.256431 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c75715-9366-48ae-8d71-c1ea2cdc0484-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:18.256486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.256465 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2pk2\" (UniqueName: \"kubernetes.io/projected/50c75715-9366-48ae-8d71-c1ea2cdc0484-kube-api-access-g2pk2\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:41:18.265288 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.265240 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh"] Oct 31 00:41:18.354679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.354651 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb"] Oct 31 00:41:18.355629 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:18.355599 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091d8d89_5935_4a95_a9c4_882e48110003.slice/crio-79f77b52f0322568f7c9d198ac31532de3027a01d285d4565a982c86a19a6029 WatchSource:0}: Error finding container 79f77b52f0322568f7c9d198ac31532de3027a01d285d4565a982c86a19a6029: Status 404 returned error can't find the container with id 79f77b52f0322568f7c9d198ac31532de3027a01d285d4565a982c86a19a6029 Oct 31 00:41:18.357392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357288 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.357392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357320 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d476266c-048c-469f-9fc6-7bad2061ea9c-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.357392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357338 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.357562 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357405 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.357657 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357634 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h9l8\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-kube-api-access-6h9l8\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.357731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357704 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.357793 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357746 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.357793 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357771 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.357898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357825 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.357955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.357928 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.358816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.358155 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d476266c-048c-469f-9fc6-7bad2061ea9c-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.358816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.358329 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.358816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.358382 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.358816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.358507 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:18.358816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.358549 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.358816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.358583 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:18.358816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.358625 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:18.360383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.360358 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.360775 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.360750 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.383213 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.383180 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.383325 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.383263 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h9l8\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-kube-api-access-6h9l8\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.558723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.558691 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:18.898936 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.893576 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh"] Oct 31 00:41:18.906629 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.906585 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:18.906803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.906673 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:18.906803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.906715 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:18.969665 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.969563 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" event={"ID":"d7474b67-7515-46f1-9b80-09c2353018a0","Type":"ContainerStarted","Data":"0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765"} Oct 31 00:41:18.974602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.974561 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" event={"ID":"d7474b67-7515-46f1-9b80-09c2353018a0","Type":"ContainerStarted","Data":"7d5782cdcb635556105d90b13ca54ccbc285911e52e7d623dd85901278152bbb"} Oct 31 00:41:18.980341 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.980304 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" event={"ID":"50c75715-9366-48ae-8d71-c1ea2cdc0484","Type":"ContainerDied","Data":"92b9769430419d5ed48717205baefd92a3ddd0bf0aa761a286e99b595bcbb05f"} Oct 31 00:41:18.980481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.980367 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b9769430419d5ed48717205baefd92a3ddd0bf0aa761a286e99b595bcbb05f" Oct 31 00:41:18.980481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.980470 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp" Oct 31 00:41:18.982252 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.982227 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" event={"ID":"d476266c-048c-469f-9fc6-7bad2061ea9c","Type":"ContainerStarted","Data":"def0b976e7ad518f5a09369fdac763dd38f819b133f28ab200683cd5bae82d92"} Oct 31 00:41:18.984260 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.984235 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" event={"ID":"091d8d89-5935-4a95-a9c4-882e48110003","Type":"ContainerStarted","Data":"a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0"} Oct 31 00:41:18.984348 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:18.984285 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" event={"ID":"091d8d89-5935-4a95-a9c4-882e48110003","Type":"ContainerStarted","Data":"79f77b52f0322568f7c9d198ac31532de3027a01d285d4565a982c86a19a6029"} Oct 31 00:41:19.065154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:19.065116 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:19.065815 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:19.065789 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:19.065965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:19.065942 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:41:19.068038 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:19.067941 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" podStartSLOduration=2.067923781 podStartE2EDuration="2.067923781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:19.014915635 +0000 UTC m=+1296.713029401" watchObservedRunningTime="2026-10-31 00:41:19.067923781 +0000 UTC m=+1296.766037549" Oct 31 00:41:19.083250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:19.083199 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" podStartSLOduration=2.083185762 podStartE2EDuration="2.083185762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:19.080563575 +0000 UTC m=+1296.778677339" watchObservedRunningTime="2026-10-31 00:41:19.083185762 +0000 UTC m=+1296.781299527" Oct 31 00:41:19.168622 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:19.168527 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:19.805076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:19.805046 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:19.806586 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:19.806561 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:41:19.806659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:19.806624 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:41:20.005553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:20.005511 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" event={"ID":"d476266c-048c-469f-9fc6-7bad2061ea9c","Type":"ContainerStarted","Data":"09de2e61761e4f118b452377ecdac5127c4d34945632547d8476172be1977c31"} Oct 31 00:41:20.053941 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:20.053898 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:20.054227 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:20.054199 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:41:20.563975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:20.563935 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:20.564755 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:20.564727 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:20.564928 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:20.564899 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:41:20.804704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:20.804668 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:41:20.804962 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:20.804932 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:41:21.066479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:21.062985 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:21.066479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:21.063054 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:41:21.118481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:21.115639 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:21.562544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:21.561922 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:21.562544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:21.561989 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:41:21.804717 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:21.804679 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:41:21.805120 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:21.804733 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:41:22.053491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:22.052983 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:22.053491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:22.053046 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:41:22.563488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:22.562920 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:22.563488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:22.562990 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:41:22.804630 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:22.804599 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:41:22.804868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:22.804843 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:41:23.048925 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:23.048887 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:23.049247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:23.049217 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:41:23.168478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:23.161627 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:23.560478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:23.560007 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:23.560478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:23.560089 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:41:23.805428 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:23.805391 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:41:23.805865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:23.805442 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:41:23.955478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:23.955240 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:23.994744 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:23.994711 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:24.036477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:24.035878 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" podStartSLOduration=6.035856743 podStartE2EDuration="6.035856743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:20.075805964 +0000 UTC m=+1297.773919729" watchObservedRunningTime="2026-10-31 00:41:24.035856743 +0000 UTC m=+1301.733970509" Oct 31 00:41:24.056479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:24.055891 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:24.056479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:24.055956 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:41:24.135480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:24.133931 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:41:24.565029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:24.564407 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:24.565029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:24.564512 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:41:24.804656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:24.804618 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:41:24.804860 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:24.804767 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:41:25.032476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:25.031443 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:25.047358 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:25.046757 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:41:25.047358 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:25.046825 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:41:25.559727 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:25.559684 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:41:25.560069 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:25.559754 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:41:25.810532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:25.810416 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:25.810906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:25.810879 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:25.811854 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:25.811835 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:41:26.051322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:26.051286 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:26.051779 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:26.051740 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:26.052555 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:26.052524 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:41:27.145317 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:27.145284 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:27.147540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:27.147516 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:27.170479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:27.166700 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 00:41:27.293267 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:27.293229 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:27.914613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:27.914561 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:28.915870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:28.915820 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:29.509303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:29.509267 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:31.201735 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:31.201693 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:32.579133 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.579096 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj"] Oct 31 00:41:32.584265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.584238 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.597195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.597162 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-daaf26-openshift-default-dockercfg-ld79m\"" Oct 31 00:41:32.611549 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.611506 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj"] Oct 31 00:41:32.658123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.658083 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.658318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.658172 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkfwq\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-kube-api-access-dkfwq\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.658318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.658226 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4826a-0872-466a-a29e-1bd82254a45d-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.658318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.658268 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4826a-0872-466a-a29e-1bd82254a45d-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.658548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.658522 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.658663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.658587 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.658663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.658620 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.658663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.658656 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.658787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.658731 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.759759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.759717 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.759975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.759876 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.759975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.759911 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.759975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.759959 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.760140 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.760008 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.760140 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.760086 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.760140 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.760111 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkfwq\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-kube-api-access-dkfwq\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.760299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.760159 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4826a-0872-466a-a29e-1bd82254a45d-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.760299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.760196 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.760299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.760222 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4826a-0872-466a-a29e-1bd82254a45d-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.760546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.760524 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.760626 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.760558 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.760686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.760669 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.761045 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.761024 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4826a-0872-466a-a29e-1bd82254a45d-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.762682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.762660 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.762815 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.762797 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4826a-0872-466a-a29e-1bd82254a45d-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.781584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.781549 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.781584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.781569 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkfwq\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-kube-api-access-dkfwq\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.903009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.902918 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:32.929848 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:32.929814 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:33.182580 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:41:33.182442 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:41:33.255985 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:33.255951 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de4826a_0872_466a_a29e_1bd82254a45d.slice/crio-b27c5611a34684268ed7511a008877e6420043763ff15aa8e543886efa731cba WatchSource:0}: Error finding container b27c5611a34684268ed7511a008877e6420043763ff15aa8e543886efa731cba: Status 404 returned error can't find the container with id b27c5611a34684268ed7511a008877e6420043763ff15aa8e543886efa731cba Oct 31 00:41:33.256546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:33.256523 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj"] Oct 31 00:41:33.258675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:33.258638 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:33.258789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:33.258722 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:33.258789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:33.258766 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:34.070594 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:34.070559 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:34.259653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:34.259613 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" event={"ID":"9de4826a-0872-466a-a29e-1bd82254a45d","Type":"ContainerStarted","Data":"474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11"} Oct 31 00:41:34.259855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:34.259682 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" event={"ID":"9de4826a-0872-466a-a29e-1bd82254a45d","Type":"ContainerStarted","Data":"b27c5611a34684268ed7511a008877e6420043763ff15aa8e543886efa731cba"} Oct 31 00:41:34.327588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:34.318979 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" podStartSLOduration=2.318959462 podStartE2EDuration="2.318959462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:34.31677611 +0000 UTC m=+1312.014889876" watchObservedRunningTime="2026-10-31 00:41:34.318959462 +0000 UTC m=+1312.017073229" Oct 31 00:41:34.892032 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:34.891978 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:34.903809 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:34.903774 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:34.905970 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:34.905946 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:41:34.906105 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:34.906010 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:41:35.903799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:35.903759 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:41:35.904219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:35.903821 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:41:36.847648 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:36.847612 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:36.904226 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:36.904173 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:41:36.904679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:36.904298 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:41:37.777803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:37.777753 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:37.904150 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:37.904112 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:41:37.904420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:37.904390 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:41:38.909479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:38.909344 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:41:38.909479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:38.909424 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:41:39.637849 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:39.637801 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:39.679311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:39.679097 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:39.904277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:39.904188 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:41:39.904463 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:39.904275 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:41:40.908643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:40.908607 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:40.909083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:40.909032 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:40.909765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:40.909744 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 00:41:41.470909 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:41.470865 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:43.124339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.124301 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:43.749232 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.749198 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj"] Oct 31 00:41:43.755787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.755758 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.768580 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.768550 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"openshift-service-ca.crt\"" Oct 31 00:41:43.778349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.778324 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"distinct-w2-735123-kserve-self-signed-certs\"" Oct 31 00:41:43.778349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.778334 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"kube-root-ca.crt\"" Oct 31 00:41:43.778349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.778329 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"default-dockercfg-t4mw9\"" Oct 31 00:41:43.782615 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.782594 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj"] Oct 31 00:41:43.848092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.848056 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:43.863233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.863199 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-dshm\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.863422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.863240 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-model-cache\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.863422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.863379 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-tmp-dir\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.863599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.863550 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaa793a8-217f-4df2-986a-fcb54b191a46-tls-certs\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.863667 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.863618 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6jq\" (UniqueName: \"kubernetes.io/projected/eaa793a8-217f-4df2-986a-fcb54b191a46-kube-api-access-6c6jq\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.863739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.863722 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-home\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.964675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.964636 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-dshm\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.964865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.964690 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-model-cache\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.964865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.964763 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-tmp-dir\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.964865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.964859 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaa793a8-217f-4df2-986a-fcb54b191a46-tls-certs\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.964991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.964887 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6jq\" (UniqueName: \"kubernetes.io/projected/eaa793a8-217f-4df2-986a-fcb54b191a46-kube-api-access-6c6jq\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.964991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.964928 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-home\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.965739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.965707 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-tmp-dir\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.966136 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.966112 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-model-cache\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.966136 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.966130 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-home\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.967367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.967345 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-dshm\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.968179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.968150 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaa793a8-217f-4df2-986a-fcb54b191a46-tls-certs\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:43.987633 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:43.987592 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6jq\" (UniqueName: \"kubernetes.io/projected/eaa793a8-217f-4df2-986a-fcb54b191a46-kube-api-access-6c6jq\") pod \"distinct-w2-735123-kserve-6664f4f65b-mk4mj\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:44.072775 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:44.072733 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:44.161495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:44.154519 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:44.380854 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:44.380820 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj"] Oct 31 00:41:44.381021 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:44.380997 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaa793a8_217f_4df2_986a_fcb54b191a46.slice/crio-ceebb569f409ae9af770daa073a6906a785d3d70e3d2c5c115ca4f74f3e7f558 WatchSource:0}: Error finding container ceebb569f409ae9af770daa073a6906a785d3d70e3d2c5c115ca4f74f3e7f558: Status 404 returned error can't find the container with id ceebb569f409ae9af770daa073a6906a785d3d70e3d2c5c115ca4f74f3e7f558 Oct 31 00:41:44.390052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:44.390011 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" event={"ID":"eaa793a8-217f-4df2-986a-fcb54b191a46","Type":"ContainerStarted","Data":"ceebb569f409ae9af770daa073a6906a785d3d70e3d2c5c115ca4f74f3e7f558"} Oct 31 00:41:44.796141 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:44.796091 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:45.399767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:45.399726 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" event={"ID":"eaa793a8-217f-4df2-986a-fcb54b191a46","Type":"ContainerStarted","Data":"35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3"} Oct 31 00:41:45.439651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:45.439590 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podStartSLOduration=2.439575052 podStartE2EDuration="2.439575052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:45.436428467 +0000 UTC m=+1323.134542246" watchObservedRunningTime="2026-10-31 00:41:45.439575052 +0000 UTC m=+1323.137688839" Oct 31 00:41:46.073640 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:46.073604 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:46.395497 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:46.395375 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:47.842164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.842132 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85"] Oct 31 00:41:47.847023 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.847001 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:47.875152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.875118 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85"] Oct 31 00:41:47.920889 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.920841 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:47.921099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.920932 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:47.921099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.920980 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:47.921099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.921042 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gb7\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-kube-api-access-p8gb7\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:47.921266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.921121 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:47.921266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.921156 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:47.921266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.921227 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:47.921416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.921282 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8ef570-9a17-4fab-823d-896f9fa99067-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:47.921416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:47.921355 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.025604 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.025510 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.025916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.025654 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.025916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.025707 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.025916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.025775 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gb7\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-kube-api-access-p8gb7\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.025916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.025855 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.025916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.025886 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.026237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.025956 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.026237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.026014 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8ef570-9a17-4fab-823d-896f9fa99067-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.026237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.026092 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.028165 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.028128 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.028746 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.028722 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.028974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.028955 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.029194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.029177 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.029830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.029787 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8ef570-9a17-4fab-823d-896f9fa99067-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.046551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.033974 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.046551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.038512 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:48.048281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.048239 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.054372 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.054339 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gb7\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-kube-api-access-p8gb7\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.054755 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.054737 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.164957 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.164861 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:48.489824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.489783 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85"] Oct 31 00:41:48.490536 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:41:48.490510 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8ef570_9a17_4fab_823d_896f9fa99067.slice/crio-e9b2d12c83a8ff6ed5c5a0a44e7fe90081280d7353794831f140a6a686460d7a WatchSource:0}: Error finding container e9b2d12c83a8ff6ed5c5a0a44e7fe90081280d7353794831f140a6a686460d7a: Status 404 returned error can't find the container with id e9b2d12c83a8ff6ed5c5a0a44e7fe90081280d7353794831f140a6a686460d7a Oct 31 00:41:48.493118 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.493077 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:48.493217 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.493161 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:48.493217 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:48.493206 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:49.445111 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:49.445073 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" event={"ID":"8e8ef570-9a17-4fab-823d-896f9fa99067","Type":"ContainerStarted","Data":"4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7"} Oct 31 00:41:49.445544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:49.445140 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" event={"ID":"8e8ef570-9a17-4fab-823d-896f9fa99067","Type":"ContainerStarted","Data":"e9b2d12c83a8ff6ed5c5a0a44e7fe90081280d7353794831f140a6a686460d7a"} Oct 31 00:41:49.496184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:49.496151 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:49.499396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:49.499335 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" podStartSLOduration=2.499316125 podStartE2EDuration="2.499316125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:49.496783797 +0000 UTC m=+1327.194897562" watchObservedRunningTime="2026-10-31 00:41:49.499316125 +0000 UTC m=+1327.197429892" Oct 31 00:41:49.902472 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:49.902417 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:50.165692 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:50.165606 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:50.167124 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:50.167102 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:41:50.167182 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:50.167152 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:41:51.166066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:51.166026 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:41:51.166501 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:51.166079 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:41:51.967951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:51.967915 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:52.165679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:52.165644 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:41:52.165844 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:52.165706 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:41:53.165488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:53.165415 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:41:53.165948 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:53.165515 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:41:53.876163 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:53.876119 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:54.073647 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:54.073603 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:54.086784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:54.086751 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:54.166315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:54.166218 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:41:54.166315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:54.166280 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:41:54.494414 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:54.494334 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:41:55.231301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:55.231263 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:55.231867 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:55.231838 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:55.234018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:55.233997 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:41:55.606750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:55.606715 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:56.041368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:56.041261 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:57.410990 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:41:57.410946 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:00.161347 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:00.161307 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:01.656193 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.656154 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx"] Oct 31 00:42:01.660768 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.660742 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.685782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.685733 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b024b5\"/\"distinct-2-w1-b024b5-kserve-self-signed-certs\"" Oct 31 00:42:01.686090 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.686055 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx"] Oct 31 00:42:01.722306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.722267 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-tmp-dir\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.722306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.722311 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b7d76f-7f3d-4f53-bc75-840933fde90a-tls-certs\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.722629 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.722567 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-model-cache\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.722695 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.722641 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-dshm\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.722768 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.722750 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-home\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.722898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.722877 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9r55\" (UniqueName: \"kubernetes.io/projected/62b7d76f-7f3d-4f53-bc75-840933fde90a-kube-api-access-p9r55\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.741756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.741679 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:01.823778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.823735 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-model-cache\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.823976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.823800 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-dshm\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.823976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.823879 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-home\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.824096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.823975 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9r55\" (UniqueName: \"kubernetes.io/projected/62b7d76f-7f3d-4f53-bc75-840933fde90a-kube-api-access-p9r55\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.824096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.824029 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-tmp-dir\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.824096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.824065 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b7d76f-7f3d-4f53-bc75-840933fde90a-tls-certs\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.824257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.824199 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-model-cache\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.824383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.824358 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-home\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.824630 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.824496 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-tmp-dir\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.826663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.826635 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-dshm\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.826797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.826741 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b7d76f-7f3d-4f53-bc75-840933fde90a-tls-certs\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.846858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.846815 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9r55\" (UniqueName: \"kubernetes.io/projected/62b7d76f-7f3d-4f53-bc75-840933fde90a-kube-api-access-p9r55\") pod \"distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:01.918127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.918022 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:01.991935 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:01.991890 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:02.331222 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.331187 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847"] Oct 31 00:42:02.336605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.336579 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.343312 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.343281 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx"] Oct 31 00:42:02.344859 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:42:02.344831 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62b7d76f_7f3d_4f53_bc75_840933fde90a.slice/crio-2f34b51c4c65b7d30446f5400d7c9784950dd6ffb5faa003aa14883b89c2387a WatchSource:0}: Error finding container 2f34b51c4c65b7d30446f5400d7c9784950dd6ffb5faa003aa14883b89c2387a: Status 404 returned error can't find the container with id 2f34b51c4c65b7d30446f5400d7c9784950dd6ffb5faa003aa14883b89c2387a Oct 31 00:42:02.363058 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.363025 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847"] Oct 31 00:42:02.439838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.439810 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxst\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-kube-api-access-rsxst\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.439973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.439873 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.439973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.439927 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.440068 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.440045 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.440122 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.440106 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.440186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.440174 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.440280 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.440268 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.440315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.440292 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.440369 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.440357 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.541429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541398 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.541429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541437 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.541693 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541527 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.541693 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541588 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxst\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-kube-api-access-rsxst\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.541693 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541634 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.541824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541763 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.541877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541826 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.541928 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541881 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.541928 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541897 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.542031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541942 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.542031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.541967 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.542031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.542002 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.542204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.542156 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.542525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.542503 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.543960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.543937 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.544279 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.544264 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.563050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.563012 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.563220 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.563100 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxst\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-kube-api-access-rsxst\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.576654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.576618 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" event={"ID":"62b7d76f-7f3d-4f53-bc75-840933fde90a","Type":"ContainerStarted","Data":"9479fe65913109e6a0a6904d8e10969b0d1d70b5af25131ed2d7950df15fbfa9"} Oct 31 00:42:02.576820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.576671 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" event={"ID":"62b7d76f-7f3d-4f53-bc75-840933fde90a","Type":"ContainerStarted","Data":"2f34b51c4c65b7d30446f5400d7c9784950dd6ffb5faa003aa14883b89c2387a"} Oct 31 00:42:02.637133 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.637026 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podStartSLOduration=1.6370126520000001 podStartE2EDuration="1.637012652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:02.63287638 +0000 UTC m=+1340.330990144" watchObservedRunningTime="2026-10-31 00:42:02.637012652 +0000 UTC m=+1340.335126440" Oct 31 00:42:02.655513 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.655484 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:02.977927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.977896 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847"] Oct 31 00:42:02.979360 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:42:02.979329 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8883f9_eb27_4d37_8f96_1dc644f9ef13.slice/crio-23e7d2afe538794a1696af3dd66aee309ec3fd0efd00ee048339bd1edab4805f WatchSource:0}: Error finding container 23e7d2afe538794a1696af3dd66aee309ec3fd0efd00ee048339bd1edab4805f: Status 404 returned error can't find the container with id 23e7d2afe538794a1696af3dd66aee309ec3fd0efd00ee048339bd1edab4805f Oct 31 00:42:02.982313 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.982276 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:02.982429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.982350 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:02.982429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.982383 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:02.991985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:02.991956 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:03.154589 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.154554 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz"] Oct 31 00:42:03.159290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.159266 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.192750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.192696 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz"] Oct 31 00:42:03.198802 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:42:03.198764 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:42:03.251979 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.251902 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.251979 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.251939 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab1607a-7d55-418b-9ff7-725222d81510-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.251979 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.251972 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.252198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.252102 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29gj\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-kube-api-access-v29gj\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.252198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.252129 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.252198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.252189 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.252297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.252230 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.252370 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.252349 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab1607a-7d55-418b-9ff7-725222d81510-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.252444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.252432 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354034 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.353997 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab1607a-7d55-418b-9ff7-725222d81510-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354212 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354121 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354212 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354147 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354212 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354179 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab1607a-7d55-418b-9ff7-725222d81510-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354358 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354216 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354358 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354332 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v29gj\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-kube-api-access-v29gj\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354597 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354370 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354597 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354445 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354597 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354511 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354616 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354687 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.354960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.354934 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.355054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.355029 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.355183 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.355031 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab1607a-7d55-418b-9ff7-725222d81510-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.356653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.356635 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab1607a-7d55-418b-9ff7-725222d81510-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.356957 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.356933 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.375551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.375520 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.376003 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.375980 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29gj\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-kube-api-access-v29gj\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.476997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.476963 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:03.586368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.586331 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" event={"ID":"cc8883f9-eb27-4d37-8f96-1dc644f9ef13","Type":"ContainerStarted","Data":"131b4c353a547a5c37f72b3cf83aedf909dd243500aed904aeedadaa68e2edfe"} Oct 31 00:42:03.586594 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.586388 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" event={"ID":"cc8883f9-eb27-4d37-8f96-1dc644f9ef13","Type":"ContainerStarted","Data":"23e7d2afe538794a1696af3dd66aee309ec3fd0efd00ee048339bd1edab4805f"} Oct 31 00:42:03.641705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.641631 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" podStartSLOduration=1.641611624 podStartE2EDuration="1.641611624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:03.637446453 +0000 UTC m=+1341.335560221" watchObservedRunningTime="2026-10-31 00:42:03.641611624 +0000 UTC m=+1341.339725390" Oct 31 00:42:03.656192 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.656126 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:03.695376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.695327 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:03.807614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.807580 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz"] Oct 31 00:42:03.808280 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:42:03.808251 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab1607a_7d55_418b_9ff7_725222d81510.slice/crio-b055b1492147f6610af7245b1aa0ab957b97522a49bd9dbc1d5a96f9a1396dc7 WatchSource:0}: Error finding container b055b1492147f6610af7245b1aa0ab957b97522a49bd9dbc1d5a96f9a1396dc7: Status 404 returned error can't find the container with id b055b1492147f6610af7245b1aa0ab957b97522a49bd9dbc1d5a96f9a1396dc7 Oct 31 00:42:03.835490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.830895 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:03.835490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.831008 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:03.835490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:03.831056 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:04.595557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.595514 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" event={"ID":"aab1607a-7d55-418b-9ff7-725222d81510","Type":"ContainerStarted","Data":"ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4"} Oct 31 00:42:04.596051 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.595570 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" event={"ID":"aab1607a-7d55-418b-9ff7-725222d81510","Type":"ContainerStarted","Data":"b055b1492147f6610af7245b1aa0ab957b97522a49bd9dbc1d5a96f9a1396dc7"} Oct 31 00:42:04.661496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.660279 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:04.661496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.660359 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:42:04.709349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.709289 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" podStartSLOduration=1.709270601 podStartE2EDuration="1.709270601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:04.652247124 +0000 UTC m=+1342.350360890" watchObservedRunningTime="2026-10-31 00:42:04.709270601 +0000 UTC m=+1342.407384426" Oct 31 00:42:04.709619 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.709503 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c"] Oct 31 00:42:04.760535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.757552 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c"] Oct 31 00:42:04.760535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.757679 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:04.772625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.772581 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:42:04.772774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.772619 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:04.778514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.777546 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036ab1d2-ml67c\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:04.778514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.777995 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036ab1d2-ml67c\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:04.778514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.778140 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj48c\" (UniqueName: \"kubernetes.io/projected/547d423a-2bbb-4909-9a26-90a2338a8560-kube-api-access-dj48c\") pod \"maas-api-revoke-sub-036ab1d2-ml67c\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:04.879137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.879054 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036ab1d2-ml67c\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:04.879137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.879095 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj48c\" (UniqueName: \"kubernetes.io/projected/547d423a-2bbb-4909-9a26-90a2338a8560-kube-api-access-dj48c\") pod \"maas-api-revoke-sub-036ab1d2-ml67c\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:04.879947 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.879811 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036ab1d2-ml67c\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:04.880080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.879974 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036ab1d2-ml67c\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:04.881714 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.881692 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036ab1d2-ml67c\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:04.900019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:04.899988 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj48c\" (UniqueName: \"kubernetes.io/projected/547d423a-2bbb-4909-9a26-90a2338a8560-kube-api-access-dj48c\") pod \"maas-api-revoke-sub-036ab1d2-ml67c\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:05.087621 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:05.087582 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:05.381087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:05.381050 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:05.455557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:05.448447 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c"] Oct 31 00:42:05.479552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:05.479439 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:05.482523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:05.482491 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:05.482772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:05.482745 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 00:42:05.604829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:05.604797 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" event={"ID":"547d423a-2bbb-4909-9a26-90a2338a8560","Type":"ContainerStarted","Data":"0b856b7fe3996e2e3de82066446635fe5e352d6f4f15602dc3262262166ac2b4"} Oct 31 00:42:05.661710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:05.661673 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:05.662008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:05.661972 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:42:06.477443 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:06.477381 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:06.478050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:06.477496 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 00:42:06.652747 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:06.645691 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" event={"ID":"547d423a-2bbb-4909-9a26-90a2338a8560","Type":"ContainerStarted","Data":"34ded859130c2ed7d3138700af1cadc835141d3b9ec299b33ab1705d8067d38d"} Oct 31 00:42:06.666990 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:06.666949 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:06.667155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:06.667015 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:42:06.694285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:06.694230 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" podStartSLOduration=2.694216974 podStartE2EDuration="2.694216974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:06.690952752 +0000 UTC m=+1344.389066518" watchObservedRunningTime="2026-10-31 00:42:06.694216974 +0000 UTC m=+1344.392330739" Oct 31 00:42:06.987008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:06.986962 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:07.478104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:07.478067 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:07.478293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:07.478121 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 00:42:07.663006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:07.662968 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:07.663685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:07.663636 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:42:07.667755 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:07.667724 5978 generic.go:356] "Generic (PLEG): container finished" podID="547d423a-2bbb-4909-9a26-90a2338a8560" containerID="34ded859130c2ed7d3138700af1cadc835141d3b9ec299b33ab1705d8067d38d" exitCode=0 Oct 31 00:42:07.667907 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:07.667816 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" event={"ID":"547d423a-2bbb-4909-9a26-90a2338a8560","Type":"ContainerDied","Data":"34ded859130c2ed7d3138700af1cadc835141d3b9ec299b33ab1705d8067d38d"} Oct 31 00:42:08.477761 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:08.477728 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:08.477992 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:08.477971 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 00:42:08.568607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:08.568559 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:08.661004 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:08.660967 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:08.661723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:08.661279 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:42:09.000057 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.000027 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:09.087239 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.087208 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-xks-ca\") pod \"547d423a-2bbb-4909-9a26-90a2338a8560\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " Oct 31 00:42:09.087447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.087433 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-openshift-ca\") pod \"547d423a-2bbb-4909-9a26-90a2338a8560\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " Oct 31 00:42:09.087518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.087498 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547d423a-2bbb-4909-9a26-90a2338a8560-kube-api-access-dj48c\" (UniqueName: \"kubernetes.io/projected/547d423a-2bbb-4909-9a26-90a2338a8560-kube-api-access-dj48c\") pod \"547d423a-2bbb-4909-9a26-90a2338a8560\" (UID: \"547d423a-2bbb-4909-9a26-90a2338a8560\") " Oct 31 00:42:09.087874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.087848 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-openshift-ca" pod "547d423a-2bbb-4909-9a26-90a2338a8560" (UID: "547d423a-2bbb-4909-9a26-90a2338a8560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:09.088054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.088040 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:09.089575 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.089543 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-xks-ca" pod "547d423a-2bbb-4909-9a26-90a2338a8560" (UID: "547d423a-2bbb-4909-9a26-90a2338a8560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:09.089664 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.089567 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547d423a-2bbb-4909-9a26-90a2338a8560-kube-api-access-dj48c" pod "547d423a-2bbb-4909-9a26-90a2338a8560" (UID: "547d423a-2bbb-4909-9a26-90a2338a8560"). InnerVolumeSpecName "kube-api-access-dj48c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:09.189303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.189264 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj48c\" (UniqueName: \"kubernetes.io/projected/547d423a-2bbb-4909-9a26-90a2338a8560-kube-api-access-dj48c\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:09.189303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.189296 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547d423a-2bbb-4909-9a26-90a2338a8560-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:09.478219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.478128 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:09.478219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.478199 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 00:42:09.656218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.656179 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:42:09.656396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.656254 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:42:09.691245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.691215 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" Oct 31 00:42:09.691418 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.691221 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c" event={"ID":"547d423a-2bbb-4909-9a26-90a2338a8560","Type":"ContainerDied","Data":"0b856b7fe3996e2e3de82066446635fe5e352d6f4f15602dc3262262166ac2b4"} Oct 31 00:42:09.691418 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:09.691345 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b856b7fe3996e2e3de82066446635fe5e352d6f4f15602dc3262262166ac2b4" Oct 31 00:42:10.207619 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:10.207575 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:10.478240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:10.478146 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 00:42:10.478240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:10.478215 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 00:42:10.662789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:10.662758 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:10.663299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:10.663272 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:10.664340 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:10.664320 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 00:42:10.740808 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:10.740707 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:11.482378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:11.482344 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:11.482921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:11.482760 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:11.483365 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:11.483344 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 00:42:11.992393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:11.992354 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:12.000206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:12.000168 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:12.006611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:12.006583 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:12.729533 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:12.729489 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:13.697271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:13.697227 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:15.401097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:15.401054 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:17.196477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.195778 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:17.459642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.459539 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:17.486683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.485549 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:17.580932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.580560 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:17.594350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.593548 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:17.638691 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.638646 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:17.692430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.692394 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk"] Oct 31 00:42:17.718833 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.718726 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:17.752613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.752571 5978 cpu_manager.go:414] "containerMap: removing container" podUID="547d423a-2bbb-4909-9a26-90a2338a8560" containerName="revoke-keys" Oct 31 00:42:17.783677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.783645 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547d423a-2bbb-4909-9a26-90a2338a8560" containerName="revoke-keys" Oct 31 00:42:17.784249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.784222 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="547d423a-2bbb-4909-9a26-90a2338a8560" containerName="revoke-keys" Oct 31 00:42:18.002545 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:17.975725 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:18.004052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.003954 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:18.045479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.044512 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk"] Oct 31 00:42:18.045479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.044571 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l"] Oct 31 00:42:18.099387 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.076739 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.107814 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.107602 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.108194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.108169 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827f550b-a8d9-4074-b34d-551ae660d799-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.108512 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.108489 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.108682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.108664 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zjr\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-kube-api-access-46zjr\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.108839 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.108821 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.109024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.109005 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.109183 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.109166 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.109327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.109309 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.109563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.109522 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827f550b-a8d9-4074-b34d-551ae660d799-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.110026 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.109990 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:18.161098 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.161066 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p"] Oct 31 00:42:18.162552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.162152 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.197475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.196317 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:18.203210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.201544 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l"] Oct 31 00:42:18.203210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.201631 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:42:18.203210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.201657 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p"] Oct 31 00:42:18.203210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.201984 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236605 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236656 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827f550b-a8d9-4074-b34d-551ae660d799-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236688 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236713 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r659g\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-kube-api-access-r659g\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236748 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de28387c-28f7-484b-b722-237c98a33d1b-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236785 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236817 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236847 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236914 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236945 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827f550b-a8d9-4074-b34d-551ae660d799-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.236990 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237028 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237083 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237119 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b0220-fcbf-41bf-8437-9b18ee3d81af-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237179 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237217 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237242 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237278 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zjr\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-kube-api-access-46zjr\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237307 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237347 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wktv\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-kube-api-access-2wktv\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237390 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237417 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237469 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237508 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de28387c-28f7-484b-b722-237c98a33d1b-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237570 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237612 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.237652 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.238107 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.246571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.239073 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827f550b-a8d9-4074-b34d-551ae660d799-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.248285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.247381 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.248758 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.248703 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.253537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.253320 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827f550b-a8d9-4074-b34d-551ae660d799-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.256287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.255691 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.260520 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.260486 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.276945 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.276850 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zjr\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-kube-api-access-46zjr\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.313974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.313928 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.338962 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.338919 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.339137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.338979 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.339137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339002 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r659g\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-kube-api-access-r659g\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.339137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339031 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de28387c-28f7-484b-b722-237c98a33d1b-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.339137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339066 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.339137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339097 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.339137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339124 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.339582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339222 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.339582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339264 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.339582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339321 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.339582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339362 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b0220-fcbf-41bf-8437-9b18ee3d81af-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.339582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339426 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.339582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339482 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.339582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339542 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wktv\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-kube-api-access-2wktv\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.339904 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339586 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.339904 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339613 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.339904 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339666 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de28387c-28f7-484b-b722-237c98a33d1b-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.339904 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.339746 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.345495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.343053 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.345495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.343998 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de28387c-28f7-484b-b722-237c98a33d1b-istiod-ca-cert\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.345495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.344233 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-credential-socket\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.345495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.344503 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.345495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.344724 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.347042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.346182 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.347042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.346480 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.347042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.346770 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-data\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.348011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.347644 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b0220-fcbf-41bf-8437-9b18ee3d81af-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.349142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.348390 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-certs\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.349142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.348754 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:42:18.414661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.412823 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wktv\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-kube-api-access-2wktv\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.414661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.414434 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.414661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.414602 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-istio-token\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.426708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.425288 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r659g\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-kube-api-access-r659g\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.435282 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.428684 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.438301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.438077 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-envoy\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.443286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.441729 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.468092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.468056 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:18.469419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.469388 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de28387c-28f7-484b-b722-237c98a33d1b-istio-podinfo\") pod \"e2e-worker-w2-735123-openshift-default-5889966f5-55w4l\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.491056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.487689 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:18.502560 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.501217 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:18.513406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.512330 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:18.513951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.513929 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:42:18.514708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.514693 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:42:18.549815 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.548805 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm"] Oct 31 00:42:18.566915 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.566883 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:18.581846 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.581817 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm"] Oct 31 00:42:18.582124 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.582111 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:42:18.582268 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.582258 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:42:18.582397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.582388 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:42:18.582626 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.582613 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.770703 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.770568 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.770902 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.770729 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.770902 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.770783 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.770902 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.770868 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.771070 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.770908 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.771070 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.770928 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.771070 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.770971 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.771243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.771093 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.771243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.771214 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhs6q\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-kube-api-access-hhs6q\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.872817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.872780 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhs6q\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-kube-api-access-hhs6q\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.872986 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.872881 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.872986 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.872977 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.873057 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873027 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.873110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873085 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.873145 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873117 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.873821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873271 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.873821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873354 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.873821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873370 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-credential-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.873821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873432 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-certs\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.873821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873489 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.874168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873874 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istiod-ca-cert\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.874168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.873991 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-socket\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.874168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.874105 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-data\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.875632 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.875604 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-envoy\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.876088 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.876046 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-podinfo\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.895412 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.894701 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-token\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.896307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.896283 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhs6q\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-kube-api-access-hhs6q\") pod \"e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.900947 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.900924 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l"] Oct 31 00:42:18.902592 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:42:18.902561 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827f550b_a8d9_4074_b34d_551ae660d799.slice/crio-2b35cd2991f37471dd122d75c679f639a3e30e5710d0af207359a9f963b95210 WatchSource:0}: Error finding container 2b35cd2991f37471dd122d75c679f639a3e30e5710d0af207359a9f963b95210: Status 404 returned error can't find the container with id 2b35cd2991f37471dd122d75c679f639a3e30e5710d0af207359a9f963b95210 Oct 31 00:42:18.902707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.902610 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:18.902707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.902689 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:18.902814 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.902731 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:18.905345 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.905314 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:18.905477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.905390 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:18.905477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.905438 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:18.905855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.905833 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk"] Oct 31 00:42:18.913874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.913752 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:18.971231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.966758 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p"] Oct 31 00:42:18.972669 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.972638 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz"] Oct 31 00:42:18.972928 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.972887 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:18.973057 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.972978 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:18.973057 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.973027 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:18.979182 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.979161 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:18.997861 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.997809 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz"] Oct 31 00:42:18.998635 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.998615 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:42:18.999634 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:18.999040 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:19.043687 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.043601 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" event={"ID":"570b0220-fcbf-41bf-8437-9b18ee3d81af","Type":"ContainerStarted","Data":"f1278e2ba44919326114fd71948c285f5da9785766b01d37746b8fbe52db5c6a"} Oct 31 00:42:19.049890 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.049836 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" event={"ID":"827f550b-a8d9-4074-b34d-551ae660d799","Type":"ContainerStarted","Data":"2b35cd2991f37471dd122d75c679f639a3e30e5710d0af207359a9f963b95210"} Oct 31 00:42:19.054157 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.054128 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" event={"ID":"de28387c-28f7-484b-b722-237c98a33d1b","Type":"ContainerStarted","Data":"366b4695b33c18d85b86442fb1944944e561372982bdff18c71de5ab61e24844"} Oct 31 00:42:19.179203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.179110 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5594c2f1-ckmqz\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.179203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.179172 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5594c2f1-ckmqz\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.179575 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.179549 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbqt\" (UniqueName: \"kubernetes.io/projected/b752a492-f6d2-4854-aa00-0eb678e3f3bd-kube-api-access-qdbqt\") pod \"maas-api-revoke-sub-5594c2f1-ckmqz\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.269444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.268245 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm"] Oct 31 00:42:19.275992 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.275948 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:19.276114 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.276052 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:19.276188 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.276112 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:19.281880 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.281855 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbqt\" (UniqueName: \"kubernetes.io/projected/b752a492-f6d2-4854-aa00-0eb678e3f3bd-kube-api-access-qdbqt\") pod \"maas-api-revoke-sub-5594c2f1-ckmqz\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.283155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.282035 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5594c2f1-ckmqz\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.283155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.282095 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5594c2f1-ckmqz\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.283155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.283101 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5594c2f1-ckmqz\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.289894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.289864 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5594c2f1-ckmqz\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.306389 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.306252 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbqt\" (UniqueName: \"kubernetes.io/projected/b752a492-f6d2-4854-aa00-0eb678e3f3bd-kube-api-access-qdbqt\") pod \"maas-api-revoke-sub-5594c2f1-ckmqz\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.322690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.322655 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:19.651052 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:42:19.651021 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb752a492_f6d2_4854_aa00_0eb678e3f3bd.slice/crio-e482a63d3fc9e7eb86db52a71094fa21751545799fb7db8b0dd098dbf4b026e7 WatchSource:0}: Error finding container e482a63d3fc9e7eb86db52a71094fa21751545799fb7db8b0dd098dbf4b026e7: Status 404 returned error can't find the container with id e482a63d3fc9e7eb86db52a71094fa21751545799fb7db8b0dd098dbf4b026e7 Oct 31 00:42:19.652855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.652816 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz"] Oct 31 00:42:19.997239 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:19.997158 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:20.096517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.096420 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" event={"ID":"570b0220-fcbf-41bf-8437-9b18ee3d81af","Type":"ContainerStarted","Data":"efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525"} Oct 31 00:42:20.101998 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.101963 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" event={"ID":"827f550b-a8d9-4074-b34d-551ae660d799","Type":"ContainerStarted","Data":"ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c"} Oct 31 00:42:20.105162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.105132 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" event={"ID":"de28387c-28f7-484b-b722-237c98a33d1b","Type":"ContainerStarted","Data":"32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca"} Oct 31 00:42:20.108287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.108223 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" event={"ID":"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8","Type":"ContainerStarted","Data":"2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162"} Oct 31 00:42:20.108603 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.108561 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" event={"ID":"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8","Type":"ContainerStarted","Data":"e53868f2a05fc59b563c985e93b6936a0b48b1e96769b10257b29d011f5de02c"} Oct 31 00:42:20.191395 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.191351 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" event={"ID":"b752a492-f6d2-4854-aa00-0eb678e3f3bd","Type":"ContainerStarted","Data":"e482a63d3fc9e7eb86db52a71094fa21751545799fb7db8b0dd098dbf4b026e7"} Oct 31 00:42:20.230476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.219712 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podStartSLOduration=3.219691954 podStartE2EDuration="3.219691954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:20.215098583 +0000 UTC m=+1357.913212353" watchObservedRunningTime="2026-10-31 00:42:20.219691954 +0000 UTC m=+1357.917805720" Oct 31 00:42:20.239532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.239470 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podStartSLOduration=3.239437025 podStartE2EDuration="3.239437025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:20.168609553 +0000 UTC m=+1357.866723319" watchObservedRunningTime="2026-10-31 00:42:20.239437025 +0000 UTC m=+1357.937550794" Oct 31 00:42:20.322601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.315614 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podStartSLOduration=3.315589968 podStartE2EDuration="3.315589968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:20.26792915 +0000 UTC m=+1357.966042918" watchObservedRunningTime="2026-10-31 00:42:20.315589968 +0000 UTC m=+1358.013703733" Oct 31 00:42:20.390020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.389939 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podStartSLOduration=2.389913292 podStartE2EDuration="2.389913292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:20.328701839 +0000 UTC m=+1358.026815607" watchObservedRunningTime="2026-10-31 00:42:20.389913292 +0000 UTC m=+1358.088027059" Oct 31 00:42:20.534898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.533373 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:20.534898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.533483 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 00:42:20.591747 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.591654 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:42:20.592018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.591989 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:42:20.655653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.655608 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:42:20.655911 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.655883 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:42:20.953400 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.953337 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:42:20.953727 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:20.953692 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 00:42:21.007733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.007286 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:21.007733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.007362 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:21.007733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.007402 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:21.007733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.007446 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:21.345571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.345533 5978 generic.go:356] "Generic (PLEG): container finished" podID="b752a492-f6d2-4854-aa00-0eb678e3f3bd" containerID="cde965b70a05192bcd53bd7a22b83256714a4be51ae0281226ef46e42aea96b9" exitCode=0 Oct 31 00:42:21.353345 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.352548 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" event={"ID":"b752a492-f6d2-4854-aa00-0eb678e3f3bd","Type":"ContainerDied","Data":"cde965b70a05192bcd53bd7a22b83256714a4be51ae0281226ef46e42aea96b9"} Oct 31 00:42:21.488916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.488879 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:21.489634 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.489602 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 00:42:21.514086 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.514050 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:42:21.514381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.514350 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:42:21.571138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.570917 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:42:21.571138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.571028 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:42:21.593106 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.591548 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:21.593106 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.591977 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:42:21.655680 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.655576 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:21.729477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.728625 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:42:21.918031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.917937 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:42:21.918272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.918242 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 00:42:21.921559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:21.921519 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:42:22.106259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.106209 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.292716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.292612 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:22.334774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.334730 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:42:22.583894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.560942 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.584492 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.584558 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.601736 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.601791 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.601872 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.601921 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.602519 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.602897 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.603208 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.603264 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:42:22.608474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.607344 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.658741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.658616 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:42:22.659427 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.659406 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:42:22.754797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.752070 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:22.754797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.752492 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:22.947702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.926561 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:42:22.976510 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.933614 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:42:22.976813 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.976772 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 00:42:22.976997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.950640 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:42:23.022480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:22.994781 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:23.141477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.122510 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:23.171478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.142224 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:42:23.371228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.362345 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:42:23.371228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.364216 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:42:23.390618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.390570 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 31 00:42:23.604903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.602638 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:23.604903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.603413 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:23.604903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.603503 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:42:23.635562 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.635061 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:23.690606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.672936 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:42:23.743485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.723867 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:42:23.743485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.715724 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:23.743485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.723965 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 00:42:23.772531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.763567 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:42:23.772531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.763642 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:42:23.780902 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.775627 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:42:23.785045 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.782983 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:42:23.785228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.784304 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:42:23.785742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.785721 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:42:23.916536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.908107 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:23.974478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.936327 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b752a492-f6d2-4854-aa00-0eb678e3f3bd-kube-api-access-qdbqt\" (UniqueName: \"kubernetes.io/projected/b752a492-f6d2-4854-aa00-0eb678e3f3bd-kube-api-access-qdbqt\") pod \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " Oct 31 00:42:23.974930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.974897 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-xks-ca\") pod \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " Oct 31 00:42:23.992482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:23.986797 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-openshift-ca\") pod \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\" (UID: \"b752a492-f6d2-4854-aa00-0eb678e3f3bd\") " Oct 31 00:42:24.058679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.058597 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-openshift-ca" pod "b752a492-f6d2-4854-aa00-0eb678e3f3bd" (UID: "b752a492-f6d2-4854-aa00-0eb678e3f3bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:24.183544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.152436 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-xks-ca" pod "b752a492-f6d2-4854-aa00-0eb678e3f3bd" (UID: "b752a492-f6d2-4854-aa00-0eb678e3f3bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:24.184308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.184267 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:42:24.237479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.211529 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:24.410117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.410071 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:24.410392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.410370 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b752a492-f6d2-4854-aa00-0eb678e3f3bd-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:24.457405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.457315 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.457714 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.457684 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 00:42:24.468170 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.468127 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:24.547901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.547847 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 31 00:42:24.553480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.499991 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.569629 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.569580 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 00:42:24.673743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.572272 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.673743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.673140 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:42:24.673743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.573772 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:42:24.673743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.673206 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:42:24.694479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.687950 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b752a492-f6d2-4854-aa00-0eb678e3f3bd-kube-api-access-qdbqt" pod "b752a492-f6d2-4854-aa00-0eb678e3f3bd" (UID: "b752a492-f6d2-4854-aa00-0eb678e3f3bd"). InnerVolumeSpecName "kube-api-access-qdbqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:24.793621 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.793445 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:42:24.793896 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.793868 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:42:24.794599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.794552 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:42:24.801232 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.801191 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:24.837508 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.822731 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 00:42:24.916386 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:24.916314 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.262295 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.259918 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdbqt\" (UniqueName: \"kubernetes.io/projected/b752a492-f6d2-4854-aa00-0eb678e3f3bd-kube-api-access-qdbqt\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:25.296611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.296555 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded" Oct 31 00:42:25.363675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.306727 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:42:25.363675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.341038 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 00:42:25.363675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.311379 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:25.363675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.344894 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:42:25.363675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.346419 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:42:25.363675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.349906 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.363675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.352110 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:42:25.394778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.394730 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.550905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.538142 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:25.550905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.538200 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 00:42:25.585525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.579924 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:42:25.585525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.579996 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:42:25.600529 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.600493 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:42:25.600815 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.600788 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:42:25.772516 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.703379 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.819566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.803683 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:42:25.819566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.807593 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:25.824199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.824129 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:42:25.824526 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.824499 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:42:25.845733 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.845685 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 00:42:25.932477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:25.921596 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.247054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.195908 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:42:26.247054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.195979 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 00:42:26.247054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.226706 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" event={"ID":"b752a492-f6d2-4854-aa00-0eb678e3f3bd","Type":"ContainerDied","Data":"e482a63d3fc9e7eb86db52a71094fa21751545799fb7db8b0dd098dbf4b026e7"} Oct 31 00:42:26.247054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.226798 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e482a63d3fc9e7eb86db52a71094fa21751545799fb7db8b0dd098dbf4b026e7" Oct 31 00:42:26.247054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.245717 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.255477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.249523 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz" Oct 31 00:42:26.371757 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.371707 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.560559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.560431 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.784475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.780575 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:42:26.817541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.809533 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:42:26.888250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.807263 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:42:26.888579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.801312 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.888732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.784029 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:42:26.888899 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.888858 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:42:26.889008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.863733 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:42:26.889113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.863754 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:42:26.889224 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.863773 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:42:26.889342 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.863694 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:26.889435 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.797957 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:26.889570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.889538 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 00:42:26.916647 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.916528 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:42:26.916647 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.916589 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 00:42:26.920002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.919969 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:42:26.938554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:26.932860 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:42:27.011228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.011198 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:42:27.178376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.178281 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:42:27.210481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.187591 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:42:27.210481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.203032 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:27.325623 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.325397 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:42:27.338072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.337107 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:42:27.348351 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.348323 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:42:27.350816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.350214 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:42:27.386576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.386546 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:42:27.404609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.404580 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:42:27.432352 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.432255 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:27.470798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.470756 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:42:27.470990 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.470823 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 00:42:27.510972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.510939 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:42:27.511288 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.511242 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:42:27.570436 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.570402 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:42:27.570852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.570817 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:42:27.925745 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.925573 5978 patch_prober.go:29] interesting pod/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:42:27.925745 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:27.925652 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 00:42:28.210077 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.209982 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:28.368527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.367494 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:28.475791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.475711 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:28.476262 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.476241 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:28.477113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.477094 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:42:28.514422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.514385 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:28.514751 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.514730 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:28.515422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.515398 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 00:42:28.576978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.576949 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:28.583546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.583512 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:28.583918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:28.583900 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:42:29.097179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:29.097140 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:29.098053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:29.098027 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:29.099137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:29.099104 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 00:42:29.559584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:29.559372 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:31.208466 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:31.208429 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:32.899651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:32.899598 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:33.012833 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.012794 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l"] Oct 31 00:42:33.013787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.013764 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b752a492-f6d2-4854-aa00-0eb678e3f3bd" containerName="revoke-keys" Oct 31 00:42:33.013787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.013785 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b752a492-f6d2-4854-aa00-0eb678e3f3bd" containerName="revoke-keys" Oct 31 00:42:33.013938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.013909 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b752a492-f6d2-4854-aa00-0eb678e3f3bd" containerName="revoke-keys" Oct 31 00:42:33.017800 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.017780 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.041272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.041240 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l"] Oct 31 00:42:33.161577 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.161483 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.161744 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.161599 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.161744 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.161661 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.161844 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.161771 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.161918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.161891 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.161991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.161974 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.162030 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.162007 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.162099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.162087 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.162138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.162123 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4fms\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-kube-api-access-n4fms\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.183390 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:42:33.183355 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:42:33.263080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263040 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263145 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263179 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263207 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263269 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263307 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4fms\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-kube-api-access-n4fms\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263376 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263434 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263510 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263579 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-certs\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263669 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.263952 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263931 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-credential-socket\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.264017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.263992 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-data\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.264017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.264007 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istiod-ca-cert\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.265761 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.265734 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-envoy\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.265925 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.265903 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-podinfo\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.284742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.284715 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-token\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.284873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.284784 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4fms\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-kube-api-access-n4fms\") pod \"e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.333697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.333664 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:33.583837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.583803 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:33.651672 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:42:33.651639 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acdf3b2_d3c6_4c7f_aab7_3ce68c0ba297.slice/crio-96f74ee3cc99d5f74608e114f84df3dc6e5179e7a27db4ee6dba0f10621103a5 WatchSource:0}: Error finding container 96f74ee3cc99d5f74608e114f84df3dc6e5179e7a27db4ee6dba0f10621103a5: Status 404 returned error can't find the container with id 96f74ee3cc99d5f74608e114f84df3dc6e5179e7a27db4ee6dba0f10621103a5 Oct 31 00:42:33.651672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.651656 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l"] Oct 31 00:42:33.654394 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.654360 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:33.654524 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.654426 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:33.654524 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:33.654502 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:34.451801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:34.451751 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" event={"ID":"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297","Type":"ContainerStarted","Data":"3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11"} Oct 31 00:42:34.452307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:34.451812 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" event={"ID":"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297","Type":"ContainerStarted","Data":"96f74ee3cc99d5f74608e114f84df3dc6e5179e7a27db4ee6dba0f10621103a5"} Oct 31 00:42:34.481813 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:34.481749 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" podStartSLOduration=2.481731712 podStartE2EDuration="2.481731712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:34.478624809 +0000 UTC m=+1372.176738576" watchObservedRunningTime="2026-10-31 00:42:34.481731712 +0000 UTC m=+1372.179845477" Oct 31 00:42:34.862480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:34.854101 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:35.336446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:35.336402 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:35.337792 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:35.337764 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:42:35.337998 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:35.337977 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:42:36.335478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:36.334989 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:42:36.335478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:36.335076 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:42:36.537494 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:36.536806 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:37.336289 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.336251 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:42:37.336975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.336936 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:42:37.802504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.802392 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk"] Oct 31 00:42:37.809431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.809400 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:37.846738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.845969 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:37.846738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.846696 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:42:37.847252 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.847218 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk"] Oct 31 00:42:37.889629 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.889590 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zk9g\" (UniqueName: \"kubernetes.io/projected/22ed1645-c162-4303-812c-86a69a125e20-kube-api-access-9zk9g\") pod \"maas-api-revoke-sub-247318e4-xjwzk\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:37.889835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.889719 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed1645-c162-4303-812c-86a69a125e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247318e4-xjwzk\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:37.889835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.889790 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed1645-c162-4303-812c-86a69a125e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247318e4-xjwzk\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:37.992092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.992051 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zk9g\" (UniqueName: \"kubernetes.io/projected/22ed1645-c162-4303-812c-86a69a125e20-kube-api-access-9zk9g\") pod \"maas-api-revoke-sub-247318e4-xjwzk\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:37.992273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.992121 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed1645-c162-4303-812c-86a69a125e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247318e4-xjwzk\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:37.992273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.992145 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed1645-c162-4303-812c-86a69a125e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247318e4-xjwzk\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:37.992895 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.992875 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed1645-c162-4303-812c-86a69a125e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247318e4-xjwzk\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:37.994720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:37.994691 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed1645-c162-4303-812c-86a69a125e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247318e4-xjwzk\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:38.038306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:38.038265 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zk9g\" (UniqueName: \"kubernetes.io/projected/22ed1645-c162-4303-812c-86a69a125e20-kube-api-access-9zk9g\") pod \"maas-api-revoke-sub-247318e4-xjwzk\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:38.158258 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:38.158220 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:38.302962 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:38.302919 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:38.359659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:38.357704 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:42:38.359659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:38.357775 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:42:38.485709 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:38.485681 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk"] Oct 31 00:42:38.485886 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:42:38.485862 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ed1645_c162_4303_812c_86a69a125e20.slice/crio-4e0ecf5ec0489ffd2426446dbc33fbeb62a687e8f5fbf721e24b799564292c3e WatchSource:0}: Error finding container 4e0ecf5ec0489ffd2426446dbc33fbeb62a687e8f5fbf721e24b799564292c3e: Status 404 returned error can't find the container with id 4e0ecf5ec0489ffd2426446dbc33fbeb62a687e8f5fbf721e24b799564292c3e Oct 31 00:42:39.335950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:39.335903 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:42:39.336170 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:39.335978 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:42:39.504259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:39.504228 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" event={"ID":"22ed1645-c162-4303-812c-86a69a125e20","Type":"ContainerStarted","Data":"0ffbd771fe134aade95b22075d553a7afcdce5e57a5081699fa583d54c802b8c"} Oct 31 00:42:39.504782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:39.504298 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" event={"ID":"22ed1645-c162-4303-812c-86a69a125e20","Type":"ContainerStarted","Data":"4e0ecf5ec0489ffd2426446dbc33fbeb62a687e8f5fbf721e24b799564292c3e"} Oct 31 00:42:39.555199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:39.555140 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" podStartSLOduration=2.555116649 podStartE2EDuration="2.555116649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:39.553501401 +0000 UTC m=+1377.251615166" watchObservedRunningTime="2026-10-31 00:42:39.555116649 +0000 UTC m=+1377.253230411" Oct 31 00:42:39.889975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:39.889879 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:40.334497 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:40.334411 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:42:40.334676 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:40.334538 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:42:40.514780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:40.514746 5978 generic.go:356] "Generic (PLEG): container finished" podID="22ed1645-c162-4303-812c-86a69a125e20" containerID="0ffbd771fe134aade95b22075d553a7afcdce5e57a5081699fa583d54c802b8c" exitCode=0 Oct 31 00:42:40.515171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:40.514848 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" event={"ID":"22ed1645-c162-4303-812c-86a69a125e20","Type":"ContainerDied","Data":"0ffbd771fe134aade95b22075d553a7afcdce5e57a5081699fa583d54c802b8c"} Oct 31 00:42:41.338821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.338788 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:41.339215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.339193 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:41.339954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.339935 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:42:41.608620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.608443 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:41.707044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.707017 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:41.778806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.778767 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ed1645-c162-4303-812c-86a69a125e20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed1645-c162-4303-812c-86a69a125e20-maas-api-openshift-ca\") pod \"22ed1645-c162-4303-812c-86a69a125e20\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " Oct 31 00:42:41.779025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.778825 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ed1645-c162-4303-812c-86a69a125e20-kube-api-access-9zk9g\" (UniqueName: \"kubernetes.io/projected/22ed1645-c162-4303-812c-86a69a125e20-kube-api-access-9zk9g\") pod \"22ed1645-c162-4303-812c-86a69a125e20\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " Oct 31 00:42:41.779025 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.778934 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22ed1645-c162-4303-812c-86a69a125e20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed1645-c162-4303-812c-86a69a125e20-maas-api-xks-ca\") pod \"22ed1645-c162-4303-812c-86a69a125e20\" (UID: \"22ed1645-c162-4303-812c-86a69a125e20\") " Oct 31 00:42:41.779283 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.779256 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ed1645-c162-4303-812c-86a69a125e20-maas-api-openshift-ca" pod "22ed1645-c162-4303-812c-86a69a125e20" (UID: "22ed1645-c162-4303-812c-86a69a125e20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:41.779713 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.779691 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed1645-c162-4303-812c-86a69a125e20-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:41.781678 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.781652 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ed1645-c162-4303-812c-86a69a125e20-maas-api-xks-ca" pod "22ed1645-c162-4303-812c-86a69a125e20" (UID: "22ed1645-c162-4303-812c-86a69a125e20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:41.781678 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.781666 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ed1645-c162-4303-812c-86a69a125e20-kube-api-access-9zk9g" pod "22ed1645-c162-4303-812c-86a69a125e20" (UID: "22ed1645-c162-4303-812c-86a69a125e20"). InnerVolumeSpecName "kube-api-access-9zk9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:41.881373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.881288 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zk9g\" (UniqueName: \"kubernetes.io/projected/22ed1645-c162-4303-812c-86a69a125e20-kube-api-access-9zk9g\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:41.881373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:41.881321 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed1645-c162-4303-812c-86a69a125e20-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:42.534087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:42.534058 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" Oct 31 00:42:42.534281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:42.534085 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk" event={"ID":"22ed1645-c162-4303-812c-86a69a125e20","Type":"ContainerDied","Data":"4e0ecf5ec0489ffd2426446dbc33fbeb62a687e8f5fbf721e24b799564292c3e"} Oct 31 00:42:42.534281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:42.534137 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e0ecf5ec0489ffd2426446dbc33fbeb62a687e8f5fbf721e24b799564292c3e" Oct 31 00:42:43.297799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:43.297742 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:44.318498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:44.318091 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:45.387694 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:45.387661 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:46.688911 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.688878 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l"] Oct 31 00:42:46.689627 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.689612 5978 cpu_manager.go:414] "containerMap: removing container" podUID="22ed1645-c162-4303-812c-86a69a125e20" containerName="revoke-keys" Oct 31 00:42:46.689675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.689628 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ed1645-c162-4303-812c-86a69a125e20" containerName="revoke-keys" Oct 31 00:42:46.689757 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.689747 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ed1645-c162-4303-812c-86a69a125e20" containerName="revoke-keys" Oct 31 00:42:46.694533 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.694513 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:46.707464 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.707423 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:46.713034 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.712997 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l"] Oct 31 00:42:46.717212 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.717180 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:42:46.791131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.791092 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d20d0b-gts9l\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:46.791356 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.791332 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-kube-api-access-8gmhj\") pod \"maas-api-revoke-sub-05d20d0b-gts9l\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:46.791485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.791438 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d20d0b-gts9l\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:46.893384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.893341 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-kube-api-access-8gmhj\") pod \"maas-api-revoke-sub-05d20d0b-gts9l\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:46.893604 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.893417 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d20d0b-gts9l\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:46.893669 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.893625 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d20d0b-gts9l\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:46.894227 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.894204 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d20d0b-gts9l\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:46.896150 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.896126 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d20d0b-gts9l\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:46.914627 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:46.914585 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-kube-api-access-8gmhj\") pod \"maas-api-revoke-sub-05d20d0b-gts9l\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:47.004664 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:47.004561 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:47.009149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:47.009115 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:47.321593 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:47.321565 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l"] Oct 31 00:42:47.321829 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:42:47.321806 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf96ab3c_a349_4ff5_b535_1653bb0ddda7.slice/crio-7eff675a2d51f9dfc56d226059b6051c2d9fbdad5d364f25c5385584f068a961 WatchSource:0}: Error finding container 7eff675a2d51f9dfc56d226059b6051c2d9fbdad5d364f25c5385584f068a961: Status 404 returned error can't find the container with id 7eff675a2d51f9dfc56d226059b6051c2d9fbdad5d364f25c5385584f068a961 Oct 31 00:42:47.580286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:47.580197 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" event={"ID":"cf96ab3c-a349-4ff5-b535-1653bb0ddda7","Type":"ContainerStarted","Data":"7eff675a2d51f9dfc56d226059b6051c2d9fbdad5d364f25c5385584f068a961"} Oct 31 00:42:48.590194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:48.590149 5978 generic.go:356] "Generic (PLEG): container finished" podID="cf96ab3c-a349-4ff5-b535-1653bb0ddda7" containerID="1629274eb20631ba751f9aca545f31a3f476e04e88de626de39258f1ab1c4352" exitCode=0 Oct 31 00:42:48.590645 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:48.590369 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" event={"ID":"cf96ab3c-a349-4ff5-b535-1653bb0ddda7","Type":"ContainerDied","Data":"1629274eb20631ba751f9aca545f31a3f476e04e88de626de39258f1ab1c4352"} Oct 31 00:42:48.597844 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:48.597817 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:48.967239 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:48.967156 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc"] Oct 31 00:42:48.971218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:48.971195 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:48.996286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:48.996257 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-735123\"/\"distinct-2-w2-735123-kserve-self-signed-certs\"" Oct 31 00:42:49.000935 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.000907 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc"] Oct 31 00:42:49.030363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.030329 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-model-cache\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.030568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.030399 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgrj\" (UniqueName: \"kubernetes.io/projected/efda8a17-abaa-4e88-ab91-3f376bdec9cb-kube-api-access-sxgrj\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.030568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.030545 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tls-certs\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.030655 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.030593 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tmp-dir\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.030694 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.030661 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-home\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.030801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.030779 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-dshm\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.067106 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.067069 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:49.132613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.132576 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-dshm\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.132811 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.132707 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-model-cache\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.132811 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.132778 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgrj\" (UniqueName: \"kubernetes.io/projected/efda8a17-abaa-4e88-ab91-3f376bdec9cb-kube-api-access-sxgrj\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.132924 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.132824 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tls-certs\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.132924 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.132889 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tmp-dir\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.133029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.132965 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-home\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.133097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.133073 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-model-cache\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.133328 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.133305 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tmp-dir\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.133398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.133311 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-home\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.134840 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.134822 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-dshm\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.135176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.135159 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tls-certs\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.153763 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.153735 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgrj\" (UniqueName: \"kubernetes.io/projected/efda8a17-abaa-4e88-ab91-3f376bdec9cb-kube-api-access-sxgrj\") pod \"distinct-2-w2-735123-kserve-54ffd9c447-mdflc\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.289429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.289334 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:49.600414 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:42:49.600380 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefda8a17_abaa_4e88_ab91_3f376bdec9cb.slice/crio-630eed6004c90a0565c4559222be1fac05be7b8a1332aa742a54fefb5d9482ff WatchSource:0}: Error finding container 630eed6004c90a0565c4559222be1fac05be7b8a1332aa742a54fefb5d9482ff: Status 404 returned error can't find the container with id 630eed6004c90a0565c4559222be1fac05be7b8a1332aa742a54fefb5d9482ff Oct 31 00:42:49.600914 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.600889 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc"] Oct 31 00:42:49.744573 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.744545 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:49.843985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.843951 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-kube-api-access-8gmhj\") pod \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " Oct 31 00:42:49.844166 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.844011 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-openshift-ca\") pod \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " Oct 31 00:42:49.844166 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.844146 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-xks-ca\") pod \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\" (UID: \"cf96ab3c-a349-4ff5-b535-1653bb0ddda7\") " Oct 31 00:42:49.844392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.844368 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-openshift-ca" pod "cf96ab3c-a349-4ff5-b535-1653bb0ddda7" (UID: "cf96ab3c-a349-4ff5-b535-1653bb0ddda7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:49.844900 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.844880 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:49.846099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.846080 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-xks-ca" pod "cf96ab3c-a349-4ff5-b535-1653bb0ddda7" (UID: "cf96ab3c-a349-4ff5-b535-1653bb0ddda7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:49.846190 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.846170 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-kube-api-access-8gmhj" pod "cf96ab3c-a349-4ff5-b535-1653bb0ddda7" (UID: "cf96ab3c-a349-4ff5-b535-1653bb0ddda7"). InnerVolumeSpecName "kube-api-access-8gmhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:49.946865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.946776 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-kube-api-access-8gmhj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:49.946865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:49.946816 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf96ab3c-a349-4ff5-b535-1653bb0ddda7-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:42:50.350786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:50.350728 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:50.610124 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:50.610043 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" Oct 31 00:42:50.610124 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:50.610074 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l" event={"ID":"cf96ab3c-a349-4ff5-b535-1653bb0ddda7","Type":"ContainerDied","Data":"7eff675a2d51f9dfc56d226059b6051c2d9fbdad5d364f25c5385584f068a961"} Oct 31 00:42:50.610124 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:50.610120 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eff675a2d51f9dfc56d226059b6051c2d9fbdad5d364f25c5385584f068a961" Oct 31 00:42:50.612544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:50.612483 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" event={"ID":"efda8a17-abaa-4e88-ab91-3f376bdec9cb","Type":"ContainerStarted","Data":"9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775"} Oct 31 00:42:50.612692 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:50.612560 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" event={"ID":"efda8a17-abaa-4e88-ab91-3f376bdec9cb","Type":"ContainerStarted","Data":"630eed6004c90a0565c4559222be1fac05be7b8a1332aa742a54fefb5d9482ff"} Oct 31 00:42:50.652820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:50.652762 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podStartSLOduration=2.652744304 podStartE2EDuration="2.652744304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:50.648438526 +0000 UTC m=+1388.346552292" watchObservedRunningTime="2026-10-31 00:42:50.652744304 +0000 UTC m=+1388.350858068" Oct 31 00:42:51.289966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:51.289925 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:52.028505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:52.028418 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:53.604750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:53.604699 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:55.164255 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:55.164218 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:55.302474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:55.301375 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:56.995702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:56.995659 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:58.776754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:58.776717 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:59.290549 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:59.290512 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:59.304049 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:59.304020 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:42:59.699142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:42:59.699112 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:43:00.424952 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:00.424908 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:02.617603 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:02.617547 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:02.711916 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:02.711709 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 00:43:02.803475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:02.802017 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:02.804855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:02.804814 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:02.805386 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:02.805359 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:43:02.850654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:02.849547 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:02.852390 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:02.852356 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.005192 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.005089 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.229492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.218228 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.531479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.521550 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 31 00:43:03.706530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.539703 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:43:03.706530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.540897 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:43:03.706530 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:43:03.613869 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:43:03.706530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.641357 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.742503 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.716317 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.835531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.826137 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.835531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.829619 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.835531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.830295 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:03.835531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.830624 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn"] Oct 31 00:43:03.835531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.831758 5978 cpu_manager.go:414] "containerMap: removing container" podUID="cf96ab3c-a349-4ff5-b535-1653bb0ddda7" containerName="revoke-keys" Oct 31 00:43:03.835531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.831778 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf96ab3c-a349-4ff5-b535-1653bb0ddda7" containerName="revoke-keys" Oct 31 00:43:03.835531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.831965 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf96ab3c-a349-4ff5-b535-1653bb0ddda7" containerName="revoke-keys" Oct 31 00:43:03.862679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.862606 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 00:43:03.864119 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:03.862933 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:04.274902 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.274807 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:43:04.394776 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.394732 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:43:04.398694 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.398651 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:43:04.403606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.403573 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l"] Oct 31 00:43:04.521479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.321104 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:43:04.544532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.413712 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:04.598925 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.598856 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:04.688699 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.687279 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:04.804607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.791873 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:43:04.805537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.805491 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:04.837674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:04.791901 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:43:05.007578 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.007521 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:43:05.217387 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.143307 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:05.217387 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.196080 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:05.309162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.308283 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:05.309162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.308370 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kl9w5" podUID="0dcb20e6-3314-47bb-983e-40d23e003f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 31 00:43:05.346161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.312973 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:43:05.346161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.313299 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:05.418602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.385610 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:05.418602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.407720 5978 patch_prober.go:29] interesting pod/thanos-querier-6db9fdb49c-5z6zr container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:43:05.418602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.407800 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-6db9fdb49c-5z6zr" podUID="2cb59038-b544-4cb1-a2af-ded4bb5d6431" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.14:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:05.458710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.458432 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn"] Oct 31 00:43:05.538523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.464189 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.641403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.641357 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.645612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.645577 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.645991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.645972 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.646180 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.646151 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.646509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.646471 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.652663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.652616 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.652999 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.652900 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.653265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.653235 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4f7c\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-kube-api-access-j4f7c\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.653442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.653416 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.689206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.689110 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:05.759956 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.759831 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.760231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.760203 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.784422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.784377 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.792895 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.792992 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793050 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4jm\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-kube-api-access-rg4jm\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793090 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793146 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793184 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793236 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793314 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793363 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793482 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4f7c\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-kube-api-access-j4f7c\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793536 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793613 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.799625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.793778 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.850799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.850760 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.907105 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.907072 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:05.907520 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.858897 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.907669 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.866664 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.950783 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.858574 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.969732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.876744 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.969958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.913155 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4f7c\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-kube-api-access-j4f7c\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.970074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.867001 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:05.970176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.961165 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974526 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l"] Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974630 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974668 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974701 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974732 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974769 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974800 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974833 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974860 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974885 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.974913 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.975803 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:43:05.988178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:05.982047 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:43:06.001496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.001405 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:06.001688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.001547 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:06.004649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.004603 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:06.012656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.010795 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr"] Oct 31 00:43:06.015559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.015530 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.015756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.015740 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.015870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.015856 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.015977 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.015965 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.016127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.016115 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4jm\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-kube-api-access-rg4jm\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.016216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.016204 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.016328 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.016315 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.016430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.016418 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.016809 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.016787 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.018779 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.018756 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-certs\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.023625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.021039 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-data\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.023625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.022032 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istiod-ca-cert\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.023625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.022555 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.023625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.022840 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-credential-socket\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.027031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.026584 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:06.044932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.044626 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:06.059023 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.058938 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr"] Oct 31 00:43:06.060044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.059319 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.091477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.075039 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:06.105421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.105385 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-envoy\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.113072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.111569 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-podinfo\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.113072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.112540 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:43:06.114007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.113808 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4jm\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-kube-api-access-rg4jm\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.118003 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.117981 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:43:06.123560 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.123541 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:43:06.140681 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.131291 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:43:06.140681 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.139094 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-token\") pod \"e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.140681 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.140591 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:43:06.149730 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.149681 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:06.150296 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.150272 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:06.226178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.225120 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55211eb4-7459-4e02-a513-087e43450b5b-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.226178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.225316 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.226178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.225417 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.226178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.225466 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.226178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.225513 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55211eb4-7459-4e02-a513-087e43450b5b-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.226178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.225613 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.226178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.225671 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p79qp\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-kube-api-access-p79qp\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.226178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.225726 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.226178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.225752 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.331481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.331086 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.331481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.331189 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.331481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.331220 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.331481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.331259 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55211eb4-7459-4e02-a513-087e43450b5b-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.331481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.331351 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.331481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.331406 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p79qp\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-kube-api-access-p79qp\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.331481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.331470 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.331944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.331497 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.331944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.331577 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55211eb4-7459-4e02-a513-087e43450b5b-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.335654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.334326 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55211eb4-7459-4e02-a513-087e43450b5b-istio-podinfo\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.339946 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.338865 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55211eb4-7459-4e02-a513-087e43450b5b-istiod-ca-cert\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.339946 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.339312 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-data\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.339946 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.339569 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-credential-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.339946 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.339802 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-socket\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.340234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.340049 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-certs\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.341905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.341859 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-envoy\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.359866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.359827 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p79qp\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-kube-api-access-p79qp\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.360859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.360804 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-istio-token\") pod \"e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.446674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.446640 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt"] Oct 31 00:43:06.454058 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.454021 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.456277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.456252 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:06.480724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.480657 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:06.481175 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.481144 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:43:06.485011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.484978 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt"] Oct 31 00:43:06.540492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.536884 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9e5c3c-g9hvt\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.540492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.537137 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9e5c3c-g9hvt\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.540492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.537570 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvq6\" (UniqueName: \"kubernetes.io/projected/6e9ddac9-2887-4aac-a8f9-85e774801b36-kube-api-access-mmvq6\") pod \"maas-api-revoke-sub-3d9e5c3c-g9hvt\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.565628 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:43:06.564757 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3dd9a6_e180_471c_8d4c_74dbfc8e1dc1.slice/crio-1857c7d3f870319056876c33a05c47b7f6a4c539ce1244801ae63e502185c84a WatchSource:0}: Error finding container 1857c7d3f870319056876c33a05c47b7f6a4c539ce1244801ae63e502185c84a: Status 404 returned error can't find the container with id 1857c7d3f870319056876c33a05c47b7f6a4c539ce1244801ae63e502185c84a Oct 31 00:43:06.579234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.577473 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:06.579234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.577609 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:06.579234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.577712 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:06.582979 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.582937 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l"] Oct 31 00:43:06.585448 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:43:06.585323 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a594534_8d06_484c_8cd1_17a4a9fe3bd2.slice/crio-51c9f12a2f757f7858598d7be25fac067f0fbef2447b455d8b3e5363736e86c0 WatchSource:0}: Error finding container 51c9f12a2f757f7858598d7be25fac067f0fbef2447b455d8b3e5363736e86c0: Status 404 returned error can't find the container with id 51c9f12a2f757f7858598d7be25fac067f0fbef2447b455d8b3e5363736e86c0 Oct 31 00:43:06.606036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.605982 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:06.606186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.606068 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:06.606186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.606106 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:06.609548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.609521 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn"] Oct 31 00:43:06.639376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.639345 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvq6\" (UniqueName: \"kubernetes.io/projected/6e9ddac9-2887-4aac-a8f9-85e774801b36-kube-api-access-mmvq6\") pod \"maas-api-revoke-sub-3d9e5c3c-g9hvt\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.640312 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.640283 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9e5c3c-g9hvt\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.640592 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.640572 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9e5c3c-g9hvt\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.648419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.647575 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9e5c3c-g9hvt\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.650996 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.648949 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9e5c3c-g9hvt\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.672108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.672053 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvq6\" (UniqueName: \"kubernetes.io/projected/6e9ddac9-2887-4aac-a8f9-85e774801b36-kube-api-access-mmvq6\") pod \"maas-api-revoke-sub-3d9e5c3c-g9hvt\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.804599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.801677 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:06.837052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.836810 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr"] Oct 31 00:43:06.845237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.842676 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:06.845237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.842749 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:06.845237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:06.842781 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:07.182333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.182250 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt"] Oct 31 00:43:07.524118 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.524079 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" event={"ID":"6e9ddac9-2887-4aac-a8f9-85e774801b36","Type":"ContainerStarted","Data":"090d3bc25e865b3f993194a7f871c0630e33ad5b00c521c0dd929b8073d7ec80"} Oct 31 00:43:07.526612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.526576 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" event={"ID":"1a594534-8d06-484c-8cd1-17a4a9fe3bd2","Type":"ContainerStarted","Data":"86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6"} Oct 31 00:43:07.526868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.526843 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" event={"ID":"1a594534-8d06-484c-8cd1-17a4a9fe3bd2","Type":"ContainerStarted","Data":"51c9f12a2f757f7858598d7be25fac067f0fbef2447b455d8b3e5363736e86c0"} Oct 31 00:43:07.592608 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.592498 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" event={"ID":"55211eb4-7459-4e02-a513-087e43450b5b","Type":"ContainerStarted","Data":"c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed"} Oct 31 00:43:07.592608 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.592567 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" event={"ID":"55211eb4-7459-4e02-a513-087e43450b5b","Type":"ContainerStarted","Data":"3548f65974234ba049fe69cd5eb3c9651b273440afa29e91330562ad27c7894a"} Oct 31 00:43:07.595432 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.595398 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" event={"ID":"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1","Type":"ContainerStarted","Data":"ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c"} Oct 31 00:43:07.595605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.595470 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" event={"ID":"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1","Type":"ContainerStarted","Data":"1857c7d3f870319056876c33a05c47b7f6a4c539ce1244801ae63e502185c84a"} Oct 31 00:43:07.639488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.637143 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podStartSLOduration=4.63712249 podStartE2EDuration="4.63712249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:07.63478468 +0000 UTC m=+1405.332898446" watchObservedRunningTime="2026-10-31 00:43:07.63712249 +0000 UTC m=+1405.335236254" Oct 31 00:43:07.699061 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:07.698996 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podStartSLOduration=4.698976839 podStartE2EDuration="4.698976839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:07.692842297 +0000 UTC m=+1405.390956063" watchObservedRunningTime="2026-10-31 00:43:07.698976839 +0000 UTC m=+1405.397090604" Oct 31 00:43:08.164536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.150778 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:08.164536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.150922 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:08.164536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.160112 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:43:08.164536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.160190 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 31 00:43:08.164536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.160624 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:43:08.164536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.160677 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 00:43:08.400171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.400064 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:08.502805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.502696 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:43:08.503287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.503250 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 00:43:08.508196 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.508165 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:08.674476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.668408 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" event={"ID":"6e9ddac9-2887-4aac-a8f9-85e774801b36","Type":"ContainerStarted","Data":"5def39e58a2e202ac6b2c128610a7a667f30536a2ccdc4f315d0c8498ce032cd"} Oct 31 00:43:08.687588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.680017 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" podStartSLOduration=5.679995159 podStartE2EDuration="5.679995159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:07.768308162 +0000 UTC m=+1405.466421931" watchObservedRunningTime="2026-10-31 00:43:08.679995159 +0000 UTC m=+1406.378108929" Oct 31 00:43:08.708568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.707566 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v"] Oct 31 00:43:08.718781 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.718748 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:08.745817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.734349 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ce6613-t5c7v\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:08.745817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.734398 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv88b\" (UniqueName: \"kubernetes.io/projected/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-kube-api-access-kv88b\") pod \"maas-api-revoke-sub-30ce6613-t5c7v\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:08.760837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.746763 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ce6613-t5c7v\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:08.795568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.783259 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v"] Oct 31 00:43:08.795568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.790952 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" podStartSLOduration=2.790931557 podStartE2EDuration="2.790931557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:08.727857087 +0000 UTC m=+1406.425970852" watchObservedRunningTime="2026-10-31 00:43:08.790931557 +0000 UTC m=+1406.489045325" Oct 31 00:43:08.896477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.888963 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ce6613-t5c7v\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:08.896477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.889157 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ce6613-t5c7v\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:08.896477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.889197 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv88b\" (UniqueName: \"kubernetes.io/projected/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-kube-api-access-kv88b\") pod \"maas-api-revoke-sub-30ce6613-t5c7v\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:08.896477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.890441 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ce6613-t5c7v\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:08.900347 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.900314 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ce6613-t5c7v\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:08.948492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:08.922132 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv88b\" (UniqueName: \"kubernetes.io/projected/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-kube-api-access-kv88b\") pod \"maas-api-revoke-sub-30ce6613-t5c7v\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:09.158478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.157885 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:43:09.158478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.157966 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 00:43:09.158784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.158679 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:43:09.158784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.158716 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 31 00:43:09.207086 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.206865 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:09.407384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.407339 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd"] Oct 31 00:43:09.444530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.425643 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.484648 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.484610 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:43:09.484909 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.484881 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 00:43:09.495504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.485667 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 00:43:09.495694 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.491711 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zgxxb\"" Oct 31 00:43:09.495815 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.491776 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 00:43:09.495913 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.491814 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 00:43:09.499419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.499391 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd"] Oct 31 00:43:09.599093 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.512041 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67469c9e-8e6c-4a36-a4e8-d85a1f010121-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.599468 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.599423 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wzm\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-kube-api-access-47wzm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.599750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.599730 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.600600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.600267 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.600600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.600313 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.600600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.600375 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.600600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.600413 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.748473 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.746094 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.771964 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.771925 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.772236 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.772212 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.772400 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.772384 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.772659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.772639 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67469c9e-8e6c-4a36-a4e8-d85a1f010121-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.781739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.772827 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wzm\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-kube-api-access-47wzm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.782044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.782017 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.806584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.778599 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.810293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.810266 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67469c9e-8e6c-4a36-a4e8-d85a1f010121-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.938882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.909579 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wzm\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-kube-api-access-47wzm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.938882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.910434 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.938882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.922127 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.938882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.922561 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:09.998145 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:09.997224 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sgntd\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:10.202475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.161994 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:43:10.202475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.162066 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 00:43:10.202475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.162932 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:43:10.202475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.162974 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 31 00:43:10.202475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.164147 5978 generic.go:356] "Generic (PLEG): container finished" podID="6e9ddac9-2887-4aac-a8f9-85e774801b36" containerID="5def39e58a2e202ac6b2c128610a7a667f30536a2ccdc4f315d0c8498ce032cd" exitCode=0 Oct 31 00:43:10.202475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.175692 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" event={"ID":"6e9ddac9-2887-4aac-a8f9-85e774801b36","Type":"ContainerDied","Data":"5def39e58a2e202ac6b2c128610a7a667f30536a2ccdc4f315d0c8498ce032cd"} Oct 31 00:43:10.203750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.203365 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:10.402479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.399981 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v"] Oct 31 00:43:10.501921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.501826 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:43:10.501921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.501887 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 00:43:10.634676 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:10.634470 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:11.205667 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:11.205635 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:43:11.205970 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:11.205936 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 31 00:43:11.237479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:11.206370 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:43:11.262658 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:11.262546 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 00:43:11.302191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:11.301052 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" event={"ID":"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac","Type":"ContainerStarted","Data":"18d5d1a39ce8dfa208e169455c7426a9162cd5aa830e9312f7e313875becc70c"} Oct 31 00:43:11.498936 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:11.498898 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:43:11.499132 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:11.498954 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 00:43:11.607527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:11.606620 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd"] Oct 31 00:43:12.009492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.009440 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:12.176539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.172483 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-openshift-ca\") pod \"6e9ddac9-2887-4aac-a8f9-85e774801b36\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " Oct 31 00:43:12.197486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.180590 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-xks-ca\") pod \"6e9ddac9-2887-4aac-a8f9-85e774801b36\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " Oct 31 00:43:12.197882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.197849 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9ddac9-2887-4aac-a8f9-85e774801b36-kube-api-access-mmvq6\" (UniqueName: \"kubernetes.io/projected/6e9ddac9-2887-4aac-a8f9-85e774801b36-kube-api-access-mmvq6\") pod \"6e9ddac9-2887-4aac-a8f9-85e774801b36\" (UID: \"6e9ddac9-2887-4aac-a8f9-85e774801b36\") " Oct 31 00:43:12.202480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.190415 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-openshift-ca" pod "6e9ddac9-2887-4aac-a8f9-85e774801b36" (UID: "6e9ddac9-2887-4aac-a8f9-85e774801b36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:12.202480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.202391 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:43:12.202710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.202443 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 31 00:43:12.206144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.206113 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:43:12.206363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.206337 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 00:43:12.207326 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.207307 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:12.210023 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.210002 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-xks-ca" pod "6e9ddac9-2887-4aac-a8f9-85e774801b36" (UID: "6e9ddac9-2887-4aac-a8f9-85e774801b36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:12.248852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.233428 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9ddac9-2887-4aac-a8f9-85e774801b36-kube-api-access-mmvq6" pod "6e9ddac9-2887-4aac-a8f9-85e774801b36" (UID: "6e9ddac9-2887-4aac-a8f9-85e774801b36"). InnerVolumeSpecName "kube-api-access-mmvq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:12.325755 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.312975 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9ddac9-2887-4aac-a8f9-85e774801b36-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:12.325755 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.313013 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvq6\" (UniqueName: \"kubernetes.io/projected/6e9ddac9-2887-4aac-a8f9-85e774801b36-kube-api-access-mmvq6\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:12.496553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.460113 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:43:12.496851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.496802 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 00:43:12.505746 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.505705 5978 generic.go:356] "Generic (PLEG): container finished" podID="7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac" containerID="bb021fd8d000b3d00f0ab92bb7a76e28913a8f152c99856467faffdcf0b542a0" exitCode=0 Oct 31 00:43:12.507628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.507608 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" Oct 31 00:43:12.508963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.508935 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" event={"ID":"67469c9e-8e6c-4a36-a4e8-d85a1f010121","Type":"ContainerStarted","Data":"4117c23d4437e7c898a09af2115b1056ed29efa71946c4a4228a00a3dd9a6b2b"} Oct 31 00:43:12.509045 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.508976 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" event={"ID":"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac","Type":"ContainerDied","Data":"bb021fd8d000b3d00f0ab92bb7a76e28913a8f152c99856467faffdcf0b542a0"} Oct 31 00:43:12.509045 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.509004 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt" event={"ID":"6e9ddac9-2887-4aac-a8f9-85e774801b36","Type":"ContainerDied","Data":"090d3bc25e865b3f993194a7f871c0630e33ad5b00c521c0dd929b8073d7ec80"} Oct 31 00:43:12.509045 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.509029 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090d3bc25e865b3f993194a7f871c0630e33ad5b00c521c0dd929b8073d7ec80" Oct 31 00:43:12.903479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:12.901491 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:13.172539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.169622 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:43:13.172539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.169689 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 00:43:13.172539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.171684 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:43:13.172539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.171734 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 31 00:43:13.459771 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.459670 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:43:13.459942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.459784 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 00:43:13.793648 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.793606 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:13.827736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.827696 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-kube-api-access-kv88b\" (UniqueName: \"kubernetes.io/projected/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-kube-api-access-kv88b\") pod \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " Oct 31 00:43:13.828008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.827850 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-xks-ca\") pod \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " Oct 31 00:43:13.828008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.827922 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-openshift-ca\") pod \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\" (UID: \"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac\") " Oct 31 00:43:13.829643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.829603 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-openshift-ca" pod "7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac" (UID: "7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:13.847577 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.847543 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-kube-api-access-kv88b" pod "7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac" (UID: "7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac"). InnerVolumeSpecName "kube-api-access-kv88b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:13.847956 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.847932 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-xks-ca" pod "7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac" (UID: "7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:13.930958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.930923 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv88b\" (UniqueName: \"kubernetes.io/projected/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-kube-api-access-kv88b\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:13.930958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.930956 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:13.931505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:13.930974 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:14.151111 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:14.151066 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:43:14.151316 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:14.151140 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 00:43:14.151316 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:14.151072 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:43:14.151316 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:14.151225 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 31 00:43:14.459529 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:14.458807 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:43:14.459529 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:14.458866 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 00:43:14.572487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:14.572037 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" event={"ID":"7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac","Type":"ContainerDied","Data":"18d5d1a39ce8dfa208e169455c7426a9162cd5aa830e9312f7e313875becc70c"} Oct 31 00:43:14.572487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:14.572107 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18d5d1a39ce8dfa208e169455c7426a9162cd5aa830e9312f7e313875becc70c" Oct 31 00:43:14.572487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:14.572182 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v" Oct 31 00:43:15.095510 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.095471 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:15.202645 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.201667 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:15.202845 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.202729 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:15.206122 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.203671 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:43:15.250679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.250224 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:15.250966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.250944 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:15.253549 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.253523 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:43:15.485710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.485633 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:15.489809 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.489746 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:15.490476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:15.490437 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:43:16.423813 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:16.423776 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:16.424224 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:16.423857 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:16.592353 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:16.592319 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" event={"ID":"67469c9e-8e6c-4a36-a4e8-d85a1f010121","Type":"ContainerStarted","Data":"c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb"} Oct 31 00:43:16.592663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:16.592643 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:16.655867 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:16.655812 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" podStartSLOduration=2.930124416 podStartE2EDuration="7.655797149s" totalImagesPullingTime="4.725672733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:16.642982756 +0000 UTC m=+1414.341096520" watchObservedRunningTime="2026-10-31 00:43:16.655797149 +0000 UTC m=+1414.353910915" Oct 31 00:43:16.974541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:16.974487 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.602885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:17.602848 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 00:43:17.959512 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:17.959431 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9"] Oct 31 00:43:17.960242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:17.960227 5978 cpu_manager.go:414] "containerMap: removing container" podUID="6e9ddac9-2887-4aac-a8f9-85e774801b36" containerName="revoke-keys" Oct 31 00:43:17.960285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:17.960245 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9ddac9-2887-4aac-a8f9-85e774801b36" containerName="revoke-keys" Oct 31 00:43:17.960285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:17.960258 5978 cpu_manager.go:414] "containerMap: removing container" podUID="7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac" containerName="revoke-keys" Oct 31 00:43:17.960285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:17.960264 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac" containerName="revoke-keys" Oct 31 00:43:17.960404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:17.960395 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9ddac9-2887-4aac-a8f9-85e774801b36" containerName="revoke-keys" Oct 31 00:43:17.960439 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:17.960405 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac" containerName="revoke-keys" Oct 31 00:43:17.964448 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:17.964423 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.006852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.006821 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9"] Oct 31 00:43:18.043649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.043616 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.043814 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.043660 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-kube-api-access-sbmf8\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.043814 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.043728 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.043814 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.043753 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.043814 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.043771 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.043951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.043821 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.043951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.043897 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.044013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.043955 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.044081 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.044057 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146170 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146128 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146180 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-kube-api-access-sbmf8\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146216 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146244 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146285 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146717 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146690 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-data\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146712 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146743 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-credential-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146892 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146821 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.146946 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146899 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.147002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.146968 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.147283 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.147260 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-certs\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.147416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.147308 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istiod-ca-cert\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.147639 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.147617 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-socket\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.148819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.148801 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-envoy\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.149725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.149709 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-podinfo\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.168087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.168051 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-kube-api-access-sbmf8\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.168697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.168677 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-token\") pod \"e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.302210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.302120 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:18.641801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.641740 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:18.642320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.641913 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:18.642320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.642007 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:18.649772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.649741 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9"] Oct 31 00:43:18.810477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:18.810132 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.648821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:19.648782 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" event={"ID":"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1","Type":"ContainerStarted","Data":"d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3"} Oct 31 00:43:19.651300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:19.648848 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" event={"ID":"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1","Type":"ContainerStarted","Data":"4ea952e210bce7b936e57af552a3b43cee7cd51c9e370a4ae4b0181cfe6339b8"} Oct 31 00:43:19.735920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:19.735852 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" podStartSLOduration=2.735832577 podStartE2EDuration="2.735832577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:19.698796475 +0000 UTC m=+1417.396910240" watchObservedRunningTime="2026-10-31 00:43:19.735832577 +0000 UTC m=+1417.433946340" Oct 31 00:43:20.302767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.302720 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:20.304369 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.304340 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:43:20.304517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.304394 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:43:20.544472 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.544100 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:20.845850 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.845812 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm"] Oct 31 00:43:20.850622 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.850599 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:20.867153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.867124 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm"] Oct 31 00:43:20.870876 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.870852 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:20.871012 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.870940 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:43:20.894817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.894781 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjvd\" (UniqueName: \"kubernetes.io/projected/13dbd9c8-fc21-4b8a-b908-3be71eea3960-kube-api-access-qrjvd\") pod \"maas-api-revoke-sub-034b1502-vflpm\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:20.895101 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.895074 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034b1502-vflpm\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:20.895211 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.895192 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034b1502-vflpm\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:20.997194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.997157 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034b1502-vflpm\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:20.997382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.997234 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034b1502-vflpm\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:20.997382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.997373 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjvd\" (UniqueName: \"kubernetes.io/projected/13dbd9c8-fc21-4b8a-b908-3be71eea3960-kube-api-access-qrjvd\") pod \"maas-api-revoke-sub-034b1502-vflpm\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:20.997956 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.997935 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034b1502-vflpm\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:20.999626 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:20.999605 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034b1502-vflpm\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:21.036785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:21.036751 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjvd\" (UniqueName: \"kubernetes.io/projected/13dbd9c8-fc21-4b8a-b908-3be71eea3960-kube-api-access-qrjvd\") pod \"maas-api-revoke-sub-034b1502-vflpm\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:21.167311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:21.167216 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:21.302961 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:21.302925 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:43:21.303157 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:21.302986 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:43:21.502988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:21.502957 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm"] Oct 31 00:43:21.503126 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:43:21.503027 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dbd9c8_fc21_4b8a_b908_3be71eea3960.slice/crio-5c4a74ca7e2390d6f672691d444e61417d45d720367e2b1ee273c90dce53b71e WatchSource:0}: Error finding container 5c4a74ca7e2390d6f672691d444e61417d45d720367e2b1ee273c90dce53b71e: Status 404 returned error can't find the container with id 5c4a74ca7e2390d6f672691d444e61417d45d720367e2b1ee273c90dce53b71e Oct 31 00:43:21.674938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:21.674900 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" event={"ID":"13dbd9c8-fc21-4b8a-b908-3be71eea3960","Type":"ContainerStarted","Data":"5c4a74ca7e2390d6f672691d444e61417d45d720367e2b1ee273c90dce53b71e"} Oct 31 00:43:22.303691 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:22.303596 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:43:22.303691 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:22.303663 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:43:22.447475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:22.447342 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:22.684914 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:22.684881 5978 generic.go:356] "Generic (PLEG): container finished" podID="13dbd9c8-fc21-4b8a-b908-3be71eea3960" containerID="cf64dfecad3b44f0b0499af112cf703175d52b9878cbf04cbbb5e99be0b431b8" exitCode=0 Oct 31 00:43:22.685108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:22.684960 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" event={"ID":"13dbd9c8-fc21-4b8a-b908-3be71eea3960","Type":"ContainerDied","Data":"cf64dfecad3b44f0b0499af112cf703175d52b9878cbf04cbbb5e99be0b431b8"} Oct 31 00:43:23.302613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:23.302582 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:43:23.302796 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:23.302633 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:43:23.898158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:23.898128 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:24.076820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.076789 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13dbd9c8-fc21-4b8a-b908-3be71eea3960-kube-api-access-qrjvd\" (UniqueName: \"kubernetes.io/projected/13dbd9c8-fc21-4b8a-b908-3be71eea3960-kube-api-access-qrjvd\") pod \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " Oct 31 00:43:24.077030 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.077000 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-openshift-ca\") pod \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " Oct 31 00:43:24.077134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.077119 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-xks-ca\") pod \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\" (UID: \"13dbd9c8-fc21-4b8a-b908-3be71eea3960\") " Oct 31 00:43:24.077369 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.077347 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-openshift-ca" pod "13dbd9c8-fc21-4b8a-b908-3be71eea3960" (UID: "13dbd9c8-fc21-4b8a-b908-3be71eea3960"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:24.077886 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.077869 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:24.079012 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.078990 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dbd9c8-fc21-4b8a-b908-3be71eea3960-kube-api-access-qrjvd" pod "13dbd9c8-fc21-4b8a-b908-3be71eea3960" (UID: "13dbd9c8-fc21-4b8a-b908-3be71eea3960"). InnerVolumeSpecName "kube-api-access-qrjvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:24.079150 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.079125 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-xks-ca" pod "13dbd9c8-fc21-4b8a-b908-3be71eea3960" (UID: "13dbd9c8-fc21-4b8a-b908-3be71eea3960"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:24.180161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.180123 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dbd9c8-fc21-4b8a-b908-3be71eea3960-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:24.180161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.180153 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrjvd\" (UniqueName: \"kubernetes.io/projected/13dbd9c8-fc21-4b8a-b908-3be71eea3960-kube-api-access-qrjvd\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:24.235477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.234467 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:24.303058 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.302985 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:43:24.303262 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.303106 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:43:24.701627 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.701598 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" Oct 31 00:43:24.701801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.701603 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm" event={"ID":"13dbd9c8-fc21-4b8a-b908-3be71eea3960","Type":"ContainerDied","Data":"5c4a74ca7e2390d6f672691d444e61417d45d720367e2b1ee273c90dce53b71e"} Oct 31 00:43:24.701801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:24.701715 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4a74ca7e2390d6f672691d444e61417d45d720367e2b1ee273c90dce53b71e" Oct 31 00:43:25.302808 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:25.302772 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:43:25.303214 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:25.302827 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:43:26.101344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:26.101307 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:26.307434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:26.307403 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:26.307906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:26.307886 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:26.308686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:26.308667 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:43:27.792110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:27.792053 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:29.409859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:29.409819 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.099060 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:31.099019 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:32.930463 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:32.930400 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.426480 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:43:33.426311 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:43:34.453476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:34.452536 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:34.828820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:34.827576 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:34.959647 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:34.959562 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Oct 31 00:43:34.973530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:34.973486 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:34.981581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:34.981538 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:34.984155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:34.984119 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.062479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.049552 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.095675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.095567 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.116806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.116762 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:43:35.122477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.120608 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.141616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.141563 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.341496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.328558 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.366664 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.366558 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Oct 31 00:43:35.597487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.596586 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:35.845508 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.843614 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:43:35.846252 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:35.846220 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:43:36.355355 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:36.329071 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:36.778550 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:36.390603 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:36.778550 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:36.390660 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 00:43:36.778550 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:36.390707 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:43:36.778550 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:36.239505 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:43:36.977786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:36.977729 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:43:37.220480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.212668 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 00:43:37.245483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.226043 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 31 00:43:37.270478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.243036 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:43:37.306984 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.243211 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.359644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.359580 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:43:37.359913 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.359881 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.596481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.568834 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:43:37.681753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.681694 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.781479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.752697 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.781479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.772940 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.796933 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.789160 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:37.815510 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.815386 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:43:37.888676 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:43:37.888582 5978 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.383s" Oct 31 00:43:37.888966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.888944 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:43:37.889299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.889281 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:43:37.889478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.889445 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:43:37.889642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.889629 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:43:37.889773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.889761 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:43:37.889903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.889892 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:43:37.890028 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.890017 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:43:37.890139 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.890128 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:43:37.890250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.890240 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:43:37.986480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:37.983181 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Oct 31 00:43:38.025614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.025563 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.249616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.246155 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 00:43:38.286536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.279561 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Oct 31 00:43:38.298793 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.298749 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.340479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.316208 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.685488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.637538 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:43:38.758606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.691960 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.889479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.852928 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 00:43:38.907973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.867648 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:43:38.937245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.937147 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:38.951559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:38.947308 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:43:39.032113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.032054 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:43:39.329690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.329643 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:39.425872 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.415870 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:39.497479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.458569 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:43:39.669949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.669836 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:39.782507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.770752 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:43:39.896366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.894603 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:39.933535 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:43:39.927447 5978 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.461s" Oct 31 00:43:39.982724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.943020 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:39.982724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.946965 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:39.982724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:39.972506 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:43:40.033325 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:40.033284 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:43:40.363479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:40.342350 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.463478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:40.450557 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.566477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:40.565566 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.661029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:40.660914 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.685633 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:40.685581 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:40.815786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:40.815738 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g"] Oct 31 00:43:40.850493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:40.823792 5978 cpu_manager.go:414] "containerMap: removing container" podUID="13dbd9c8-fc21-4b8a-b908-3be71eea3960" containerName="revoke-keys" Oct 31 00:43:41.070603 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.070570 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13dbd9c8-fc21-4b8a-b908-3be71eea3960" containerName="revoke-keys" Oct 31 00:43:41.140475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.140093 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="13dbd9c8-fc21-4b8a-b908-3be71eea3960" containerName="revoke-keys" Oct 31 00:43:41.246908 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.169755 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 00:43:41.260174 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.136333 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.282473 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.058410 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.282473 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.211568 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.292986 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.005566 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.293645 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.049609 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:43:41.297843 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.237679 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:41.411303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.411214 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:43:41.546894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.423952 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:43:41.546894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.424675 5978 patch_prober.go:29] interesting pod/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:43:41.546894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.526034 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.564238 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.564085 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:41.587785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.566591 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxfr\" (UniqueName: \"kubernetes.io/projected/c7c006ff-516f-445c-8c98-62927442b391-kube-api-access-5rxfr\") pod \"maas-api-revoke-sub-889dfd10-mph6g\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:41.587785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.566882 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c006ff-516f-445c-8c98-62927442b391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-889dfd10-mph6g\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:41.587785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.567113 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c006ff-516f-445c-8c98-62927442b391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-889dfd10-mph6g\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:41.700598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.671097 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.762389 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.740261 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.762389 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.638652 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:41.807393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.799623 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c006ff-516f-445c-8c98-62927442b391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-889dfd10-mph6g\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:41.807393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.799786 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxfr\" (UniqueName: \"kubernetes.io/projected/c7c006ff-516f-445c-8c98-62927442b391-kube-api-access-5rxfr\") pod \"maas-api-revoke-sub-889dfd10-mph6g\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:41.807393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.800557 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c006ff-516f-445c-8c98-62927442b391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-889dfd10-mph6g\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:41.846092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.834213 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g"] Oct 31 00:43:41.884488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:41.871156 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c006ff-516f-445c-8c98-62927442b391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-889dfd10-mph6g\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:42.038527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.032763 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c006ff-516f-445c-8c98-62927442b391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-889dfd10-mph6g\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:42.042507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.033749 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" podUID="a3a55405-5cba-4424-b346-9b2c8d98b3a0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:42.042507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.038946 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:43:42.059127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.059087 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:43:42.061947 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.061880 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxfr\" (UniqueName: \"kubernetes.io/projected/c7c006ff-516f-445c-8c98-62927442b391-kube-api-access-5rxfr\") pod \"maas-api-revoke-sub-889dfd10-mph6g\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:42.099072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.099042 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:42.104810 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.102541 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:43:42.106642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.105954 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:43:42.162096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.159901 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:43:42.162096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.161098 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:43:42.165440 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.165417 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:43:42.166604 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.166585 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:43:42.168366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.168350 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:43:42.174686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.171749 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:43:42.497697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.497660 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g"] Oct 31 00:43:42.707903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:42.707872 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:43.180334 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.180238 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" event={"ID":"c7c006ff-516f-445c-8c98-62927442b391","Type":"ContainerStarted","Data":"3358b8673ccaf34001141f6eb41b504a8d1cc55906c7de094dacacb39423cebd"} Oct 31 00:43:43.180334 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.180289 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" event={"ID":"c7c006ff-516f-445c-8c98-62927442b391","Type":"ContainerStarted","Data":"ed522e8dc980c4e6921575b1f58f525d81ce8c441b5ca5a77b1d829a64ef2d34"} Oct 31 00:43:43.223304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.223240 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" podStartSLOduration=4.223221158 podStartE2EDuration="4.223221158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:43.217874544 +0000 UTC m=+1440.915988311" watchObservedRunningTime="2026-10-31 00:43:43.223221158 +0000 UTC m=+1440.921334926" Oct 31 00:43:43.504065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.503986 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2"] Oct 31 00:43:43.508812 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.508789 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.532154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.532117 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2"] Oct 31 00:43:43.576869 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.576835 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntrd\" (UniqueName: \"kubernetes.io/projected/9a0d28cf-4936-4a92-b06b-b2f2f6499229-kube-api-access-cntrd\") pod \"maas-api-revoke-sub-b0c14659-qb8k2\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.577047 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.576898 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c14659-qb8k2\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.577047 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.576973 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c14659-qb8k2\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.678952 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.678920 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntrd\" (UniqueName: \"kubernetes.io/projected/9a0d28cf-4936-4a92-b06b-b2f2f6499229-kube-api-access-cntrd\") pod \"maas-api-revoke-sub-b0c14659-qb8k2\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.679117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.678984 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c14659-qb8k2\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.679117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.679026 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c14659-qb8k2\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.679588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.679566 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c14659-qb8k2\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.681378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.681356 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c14659-qb8k2\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.718626 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.718594 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntrd\" (UniqueName: \"kubernetes.io/projected/9a0d28cf-4936-4a92-b06b-b2f2f6499229-kube-api-access-cntrd\") pod \"maas-api-revoke-sub-b0c14659-qb8k2\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:43.829123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:43.829087 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:44.140760 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:43:44.140728 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0d28cf_4936_4a92_b06b_b2f2f6499229.slice/crio-aad4965eb7a88a8c2b43cabfa3667e125101c1f7a33b5ab97c597e548ec624bf WatchSource:0}: Error finding container aad4965eb7a88a8c2b43cabfa3667e125101c1f7a33b5ab97c597e548ec624bf: Status 404 returned error can't find the container with id aad4965eb7a88a8c2b43cabfa3667e125101c1f7a33b5ab97c597e548ec624bf Oct 31 00:43:44.143131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:44.143076 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2"] Oct 31 00:43:44.189284 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:44.189247 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" event={"ID":"9a0d28cf-4936-4a92-b06b-b2f2f6499229","Type":"ContainerStarted","Data":"aad4965eb7a88a8c2b43cabfa3667e125101c1f7a33b5ab97c597e548ec624bf"} Oct 31 00:43:44.191013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:44.190987 5978 generic.go:356] "Generic (PLEG): container finished" podID="c7c006ff-516f-445c-8c98-62927442b391" containerID="3358b8673ccaf34001141f6eb41b504a8d1cc55906c7de094dacacb39423cebd" exitCode=0 Oct 31 00:43:44.191125 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:44.191034 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" event={"ID":"c7c006ff-516f-445c-8c98-62927442b391","Type":"ContainerDied","Data":"3358b8673ccaf34001141f6eb41b504a8d1cc55906c7de094dacacb39423cebd"} Oct 31 00:43:44.439734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:44.439702 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:44.893652 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:44.893171 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:45.202373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.202335 5978 generic.go:356] "Generic (PLEG): container finished" podID="9a0d28cf-4936-4a92-b06b-b2f2f6499229" containerID="1135d2ea864d6da16dce6245098b6cc6d190e8f6119af94f082ba5ee6b3bcc97" exitCode=0 Oct 31 00:43:45.202943 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.202378 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" event={"ID":"9a0d28cf-4936-4a92-b06b-b2f2f6499229","Type":"ContainerDied","Data":"1135d2ea864d6da16dce6245098b6cc6d190e8f6119af94f082ba5ee6b3bcc97"} Oct 31 00:43:45.376465 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.376427 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:45.408399 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.408341 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c006ff-516f-445c-8c98-62927442b391-kube-api-access-5rxfr\" (UniqueName: \"kubernetes.io/projected/c7c006ff-516f-445c-8c98-62927442b391-kube-api-access-5rxfr\") pod \"c7c006ff-516f-445c-8c98-62927442b391\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " Oct 31 00:43:45.408627 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.408413 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c006ff-516f-445c-8c98-62927442b391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c006ff-516f-445c-8c98-62927442b391-maas-api-xks-ca\") pod \"c7c006ff-516f-445c-8c98-62927442b391\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " Oct 31 00:43:45.408707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.408641 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c006ff-516f-445c-8c98-62927442b391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c006ff-516f-445c-8c98-62927442b391-maas-api-openshift-ca\") pod \"c7c006ff-516f-445c-8c98-62927442b391\" (UID: \"c7c006ff-516f-445c-8c98-62927442b391\") " Oct 31 00:43:45.409071 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.409041 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c006ff-516f-445c-8c98-62927442b391-maas-api-openshift-ca" pod "c7c006ff-516f-445c-8c98-62927442b391" (UID: "c7c006ff-516f-445c-8c98-62927442b391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:45.409519 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.409498 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c006ff-516f-445c-8c98-62927442b391-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:45.410485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.410442 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c006ff-516f-445c-8c98-62927442b391-maas-api-xks-ca" pod "c7c006ff-516f-445c-8c98-62927442b391" (UID: "c7c006ff-516f-445c-8c98-62927442b391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:45.410616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.410596 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c006ff-516f-445c-8c98-62927442b391-kube-api-access-5rxfr" pod "c7c006ff-516f-445c-8c98-62927442b391" (UID: "c7c006ff-516f-445c-8c98-62927442b391"). InnerVolumeSpecName "kube-api-access-5rxfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:45.511485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.511438 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxfr\" (UniqueName: \"kubernetes.io/projected/c7c006ff-516f-445c-8c98-62927442b391-kube-api-access-5rxfr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:45.511485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:45.511482 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c006ff-516f-445c-8c98-62927442b391-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:46.223659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.216408 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" Oct 31 00:43:46.223659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.216439 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g" event={"ID":"c7c006ff-516f-445c-8c98-62927442b391","Type":"ContainerDied","Data":"ed522e8dc980c4e6921575b1f58f525d81ce8c441b5ca5a77b1d829a64ef2d34"} Oct 31 00:43:46.223659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.216553 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed522e8dc980c4e6921575b1f58f525d81ce8c441b5ca5a77b1d829a64ef2d34" Oct 31 00:43:46.401288 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.401262 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:46.532049 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.531963 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0d28cf-4936-4a92-b06b-b2f2f6499229-kube-api-access-cntrd\" (UniqueName: \"kubernetes.io/projected/9a0d28cf-4936-4a92-b06b-b2f2f6499229-kube-api-access-cntrd\") pod \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " Oct 31 00:43:46.532220 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.532067 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-xks-ca\") pod \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " Oct 31 00:43:46.532337 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.532316 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-openshift-ca\") pod \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\" (UID: \"9a0d28cf-4936-4a92-b06b-b2f2f6499229\") " Oct 31 00:43:46.532746 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.532716 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-openshift-ca" pod "9a0d28cf-4936-4a92-b06b-b2f2f6499229" (UID: "9a0d28cf-4936-4a92-b06b-b2f2f6499229"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:46.533811 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.533768 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:46.534879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.534842 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0d28cf-4936-4a92-b06b-b2f2f6499229-kube-api-access-cntrd" pod "9a0d28cf-4936-4a92-b06b-b2f2f6499229" (UID: "9a0d28cf-4936-4a92-b06b-b2f2f6499229"). InnerVolumeSpecName "kube-api-access-cntrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:46.556666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.554339 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-xks-ca" pod "9a0d28cf-4936-4a92-b06b-b2f2f6499229" (UID: "9a0d28cf-4936-4a92-b06b-b2f2f6499229"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:46.598190 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.598151 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:46.635404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.635358 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cntrd\" (UniqueName: \"kubernetes.io/projected/9a0d28cf-4936-4a92-b06b-b2f2f6499229-kube-api-access-cntrd\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:46.635404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:46.635402 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0d28cf-4936-4a92-b06b-b2f2f6499229-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:43:47.226190 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.226156 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" Oct 31 00:43:47.226700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.226194 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2" event={"ID":"9a0d28cf-4936-4a92-b06b-b2f2f6499229","Type":"ContainerDied","Data":"aad4965eb7a88a8c2b43cabfa3667e125101c1f7a33b5ab97c597e548ec624bf"} Oct 31 00:43:47.226700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.226374 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad4965eb7a88a8c2b43cabfa3667e125101c1f7a33b5ab97c597e548ec624bf" Oct 31 00:43:47.691322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.691291 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth"] Oct 31 00:43:47.692434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.692407 5978 cpu_manager.go:414] "containerMap: removing container" podUID="9a0d28cf-4936-4a92-b06b-b2f2f6499229" containerName="revoke-keys" Oct 31 00:43:47.692434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.692432 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0d28cf-4936-4a92-b06b-b2f2f6499229" containerName="revoke-keys" Oct 31 00:43:47.692610 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.692534 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c7c006ff-516f-445c-8c98-62927442b391" containerName="revoke-keys" Oct 31 00:43:47.692610 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.692545 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c006ff-516f-445c-8c98-62927442b391" containerName="revoke-keys" Oct 31 00:43:47.692723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.692707 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0d28cf-4936-4a92-b06b-b2f2f6499229" containerName="revoke-keys" Oct 31 00:43:47.692757 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.692740 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c006ff-516f-445c-8c98-62927442b391" containerName="revoke-keys" Oct 31 00:43:47.697291 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.697270 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.741264 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.741216 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth"] Oct 31 00:43:47.755841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.755795 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b37e6d-25c3-4b3d-975b-496139791738-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.756005 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.755977 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.756112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.756095 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26hs\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-kube-api-access-s26hs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.756221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.756198 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.756352 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.756333 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.756461 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.756424 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.756562 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.756537 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.756697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.756582 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.756785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.756694 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b37e6d-25c3-4b3d-975b-496139791738-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.857919 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.857884 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.857931 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.857969 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.857987 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.858055 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b37e6d-25c3-4b3d-975b-496139791738-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858251 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.858119 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b37e6d-25c3-4b3d-975b-496139791738-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858251 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.858222 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.858292 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-credential-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.858308 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26hs\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-kube-api-access-s26hs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.858339 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-socket\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.858363 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858706 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.858683 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-certs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.858760 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.858709 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-data\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.859034 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.859015 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b37e6d-25c3-4b3d-975b-496139791738-istiod-ca-cert\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.860354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.860332 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-envoy\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.860635 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.860614 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b37e6d-25c3-4b3d-975b-496139791738-istio-podinfo\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.879299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.879273 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-istio-token\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:47.879415 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:47.879397 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26hs\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-kube-api-access-s26hs\") pod \"e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:48.014699 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:48.014608 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:48.326633 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:48.326512 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:48.337853 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:48.337763 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth"] Oct 31 00:43:48.347483 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:43:48.347420 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b37e6d_25c3_4b3d_975b_496139791738.slice/crio-90a14bf49b78e210cb0b1dd1a54a15fc09b69b30f7090cb32e41f1c67c6a2710 WatchSource:0}: Error finding container 90a14bf49b78e210cb0b1dd1a54a15fc09b69b30f7090cb32e41f1c67c6a2710: Status 404 returned error can't find the container with id 90a14bf49b78e210cb0b1dd1a54a15fc09b69b30f7090cb32e41f1c67c6a2710 Oct 31 00:43:48.354530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:48.351713 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:48.354530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:48.351816 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:48.354530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:48.351860 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:49.249171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:49.249134 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" event={"ID":"a8b37e6d-25c3-4b3d-975b-496139791738","Type":"ContainerStarted","Data":"3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125"} Oct 31 00:43:49.249383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:49.249192 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" event={"ID":"a8b37e6d-25c3-4b3d-975b-496139791738","Type":"ContainerStarted","Data":"90a14bf49b78e210cb0b1dd1a54a15fc09b69b30f7090cb32e41f1c67c6a2710"} Oct 31 00:43:49.338481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:49.327638 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" podStartSLOduration=2.327614953 podStartE2EDuration="2.327614953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:49.29860113 +0000 UTC m=+1446.996714897" watchObservedRunningTime="2026-10-31 00:43:49.327614953 +0000 UTC m=+1447.025728719" Oct 31 00:43:50.033477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:50.032826 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:50.036033 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:50.035658 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:43:50.036033 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:50.035738 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 00:43:50.139344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:50.139304 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:51.035479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:51.034002 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:43:51.035479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:51.034087 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 00:43:51.939277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:51.939235 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:52.016926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:52.016894 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:43:52.017200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:52.017171 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 00:43:53.033478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:53.026937 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:43:53.033478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:53.027019 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 00:43:53.850440 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:53.850362 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:54.020549 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:54.015911 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:43:54.020549 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:54.015984 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 00:43:55.015371 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:55.015021 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:43:55.015371 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:55.015099 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 00:43:55.638426 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:55.638389 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:56.020261 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:56.020169 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:56.020684 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:56.020597 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:56.021276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:56.021256 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:43:57.431307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:57.431229 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:57.977153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:57.977116 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:58.144090 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:58.144048 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:58.708768 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:58.708736 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:59.292113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:43:59.292067 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:00.286301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.286265 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l"] Oct 31 00:44:00.294779 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.294754 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.323314 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.323279 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:00.323551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.323506 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:44:00.327820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.327792 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l"] Oct 31 00:44:00.395250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.395191 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e01341-02df-4bf5-8935-4309dde93168-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f689cbc1-l8r2l\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.395428 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.395285 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e01341-02df-4bf5-8935-4309dde93168-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f689cbc1-l8r2l\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.395496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.395443 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/67e01341-02df-4bf5-8935-4309dde93168-kube-api-access-7zz8w\") pod \"maas-api-revoke-sub-f689cbc1-l8r2l\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.497145 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.497112 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/67e01341-02df-4bf5-8935-4309dde93168-kube-api-access-7zz8w\") pod \"maas-api-revoke-sub-f689cbc1-l8r2l\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.497329 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.497311 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e01341-02df-4bf5-8935-4309dde93168-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f689cbc1-l8r2l\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.497423 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.497407 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e01341-02df-4bf5-8935-4309dde93168-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f689cbc1-l8r2l\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.497965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.497939 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e01341-02df-4bf5-8935-4309dde93168-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f689cbc1-l8r2l\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.499873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.499833 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e01341-02df-4bf5-8935-4309dde93168-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f689cbc1-l8r2l\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.518988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.518951 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/67e01341-02df-4bf5-8935-4309dde93168-kube-api-access-7zz8w\") pod \"maas-api-revoke-sub-f689cbc1-l8r2l\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.621419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.621376 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:00.898597 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.898442 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:00.938603 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:00.938571 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l"] Oct 31 00:44:01.371747 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:01.371711 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" event={"ID":"67e01341-02df-4bf5-8935-4309dde93168","Type":"ContainerStarted","Data":"af9790a3dae89d6493a2b0eaf5feb76387b4c5cbb439cef84eebe63f85fe9422"} Oct 31 00:44:01.372245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:01.371766 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" event={"ID":"67e01341-02df-4bf5-8935-4309dde93168","Type":"ContainerStarted","Data":"d3ac515542a3d2a2b66a65117c13cfe5f188686d218567026a4e96ae4f189d55"} Oct 31 00:44:01.413756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:01.413698 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" podStartSLOduration=1.413681223 podStartE2EDuration="1.413681223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:01.408574784 +0000 UTC m=+1459.106688549" watchObservedRunningTime="2026-10-31 00:44:01.413681223 +0000 UTC m=+1459.111794987" Oct 31 00:44:02.384274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:02.384237 5978 generic.go:356] "Generic (PLEG): container finished" podID="67e01341-02df-4bf5-8935-4309dde93168" containerID="af9790a3dae89d6493a2b0eaf5feb76387b4c5cbb439cef84eebe63f85fe9422" exitCode=0 Oct 31 00:44:02.384763 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:02.384316 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" event={"ID":"67e01341-02df-4bf5-8935-4309dde93168","Type":"ContainerDied","Data":"af9790a3dae89d6493a2b0eaf5feb76387b4c5cbb439cef84eebe63f85fe9422"} Oct 31 00:44:02.725778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:02.725646 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:03.190831 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:44:03.190789 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:44:03.582202 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.582172 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:03.657959 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.657924 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e01341-02df-4bf5-8935-4309dde93168-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e01341-02df-4bf5-8935-4309dde93168-maas-api-openshift-ca\") pod \"67e01341-02df-4bf5-8935-4309dde93168\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " Oct 31 00:44:03.658150 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.658006 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e01341-02df-4bf5-8935-4309dde93168-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e01341-02df-4bf5-8935-4309dde93168-maas-api-xks-ca\") pod \"67e01341-02df-4bf5-8935-4309dde93168\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " Oct 31 00:44:03.658219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.658167 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e01341-02df-4bf5-8935-4309dde93168-kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/67e01341-02df-4bf5-8935-4309dde93168-kube-api-access-7zz8w\") pod \"67e01341-02df-4bf5-8935-4309dde93168\" (UID: \"67e01341-02df-4bf5-8935-4309dde93168\") " Oct 31 00:44:03.658349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.658320 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e01341-02df-4bf5-8935-4309dde93168-maas-api-openshift-ca" pod "67e01341-02df-4bf5-8935-4309dde93168" (UID: "67e01341-02df-4bf5-8935-4309dde93168"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:03.658885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.658864 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e01341-02df-4bf5-8935-4309dde93168-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:03.660353 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.660325 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e01341-02df-4bf5-8935-4309dde93168-maas-api-xks-ca" pod "67e01341-02df-4bf5-8935-4309dde93168" (UID: "67e01341-02df-4bf5-8935-4309dde93168"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:03.660482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.660398 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e01341-02df-4bf5-8935-4309dde93168-kube-api-access-7zz8w" pod "67e01341-02df-4bf5-8935-4309dde93168" (UID: "67e01341-02df-4bf5-8935-4309dde93168"). InnerVolumeSpecName "kube-api-access-7zz8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:03.760350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.760247 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e01341-02df-4bf5-8935-4309dde93168-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:03.760350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:03.760283 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zz8w\" (UniqueName: \"kubernetes.io/projected/67e01341-02df-4bf5-8935-4309dde93168-kube-api-access-7zz8w\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:04.403485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:04.403412 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" event={"ID":"67e01341-02df-4bf5-8935-4309dde93168","Type":"ContainerDied","Data":"d3ac515542a3d2a2b66a65117c13cfe5f188686d218567026a4e96ae4f189d55"} Oct 31 00:44:04.403649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:04.403423 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l" Oct 31 00:44:04.403649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:04.403526 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3ac515542a3d2a2b66a65117c13cfe5f188686d218567026a4e96ae4f189d55" Oct 31 00:44:04.598595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:04.598547 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:06.432480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:06.432061 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:06.612480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:06.611525 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:06.794543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:06.783555 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.794543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:06.785948 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:06.909604 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:06.909541 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.940601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:06.940546 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:44:06.947544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:06.946704 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.962140 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:06.962096 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:06.962518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:06.962491 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.036492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.035522 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.148477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.147586 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:07.148477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.147656 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.347480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.341533 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:07.347480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.346621 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.347480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.347018 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" podUID="a3a55405-5cba-4424-b346-9b2c8d98b3a0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.427655 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.427556 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.581477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.580558 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.647481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.642192 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.822571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.812073 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:44:07.836478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.831322 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:44:07.965614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.965561 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:07.989529 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:07.966653 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:44:08.069493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.035551 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.069493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.060209 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:44:08.097953 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.073110 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:44:08.161910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.140714 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.161910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.159962 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:44:08.161910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.160062 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:44:08.199810 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.199752 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.203003 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.202560 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:44:08.326595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.326473 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:08.343883 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.342502 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.369654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.344583 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:44:08.369654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.355191 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:08.369654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.360626 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:44:08.972479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.969168 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.122531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.969714 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 00:44:09.160564 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.976973 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.225540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:08.977515 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:44:09.245480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:09.098665 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.245480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:09.241799 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:09.253722 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:09.098754 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:44:09.429482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:09.403704 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:44:09.500541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:09.431763 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:09.500541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:09.473430 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:44:09.570478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:09.556929 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:10.165478 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:44:10.110146 5978 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.572s" Oct 31 00:44:10.165478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.110291 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:44:10.165478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.110340 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:44:10.189486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.166471 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.189486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.166759 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.189486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.180534 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.249931 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.239543 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.249931 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.239932 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:44:10.250307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.250269 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.289185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.288362 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.396778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.296925 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.778488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.704636 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:10.838859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.709289 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:10.840550 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:10.709346 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:44:11.166878 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.166827 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.192799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.192752 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:44:11.194989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.194948 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.315758 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.315714 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 00:44:11.320041 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.316063 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.322977 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.316102 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:44:11.335171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.316125 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:44:11.529228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.522714 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:44:11.578518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.577191 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp"] Oct 31 00:44:11.578518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.578414 5978 cpu_manager.go:414] "containerMap: removing container" podUID="67e01341-02df-4bf5-8935-4309dde93168" containerName="revoke-keys" Oct 31 00:44:11.578518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.578439 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e01341-02df-4bf5-8935-4309dde93168" containerName="revoke-keys" Oct 31 00:44:11.607480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.583661 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e01341-02df-4bf5-8935-4309dde93168" containerName="revoke-keys" Oct 31 00:44:11.616484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.609719 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:44:11.616484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.610054 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:11.634839 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.634803 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:11.636151 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.636100 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:44:11.680546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.680482 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp"] Oct 31 00:44:11.775306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.773860 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45f4a89b-2hvqp\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:11.775306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.774148 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8nhd\" (UniqueName: \"kubernetes.io/projected/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-kube-api-access-w8nhd\") pod \"maas-api-revoke-sub-45f4a89b-2hvqp\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:11.775306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.774244 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45f4a89b-2hvqp\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:11.820727 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.820699 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:44:11.824918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.824890 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:44:11.842593 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.842538 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:11.876511 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.876437 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8nhd\" (UniqueName: \"kubernetes.io/projected/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-kube-api-access-w8nhd\") pod \"maas-api-revoke-sub-45f4a89b-2hvqp\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:11.876704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.876592 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45f4a89b-2hvqp\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:11.876865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.876837 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45f4a89b-2hvqp\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:11.877704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.877678 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45f4a89b-2hvqp\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:11.888102 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.880745 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45f4a89b-2hvqp\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:11.897473 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.895086 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:44:11.910712 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.909923 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:44:11.935517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.922936 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:44:11.935517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.923611 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:44:11.935517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.927620 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:44:11.935517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.930873 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:44:11.936163 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:11.936136 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8nhd\" (UniqueName: \"kubernetes.io/projected/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-kube-api-access-w8nhd\") pod \"maas-api-revoke-sub-45f4a89b-2hvqp\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:12.007816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:12.007772 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:12.355659 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:44:12.355599 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b2e26b_70ad_49f2_a3b8_94df8d5aeb4f.slice/crio-a1c20745d789fe1915e4ff34a0f5d164e6f3f6408764f427a0d95d5ad410f135 WatchSource:0}: Error finding container a1c20745d789fe1915e4ff34a0f5d164e6f3f6408764f427a0d95d5ad410f135: Status 404 returned error can't find the container with id a1c20745d789fe1915e4ff34a0f5d164e6f3f6408764f427a0d95d5ad410f135 Oct 31 00:44:12.357789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:12.357759 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp"] Oct 31 00:44:13.362052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:13.362011 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" event={"ID":"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f","Type":"ContainerStarted","Data":"beca1ccadc12bc772cd8ef0f29f587c0c587f9f482ce3f04c9061c0e3fcc8db8"} Oct 31 00:44:13.362446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:13.362068 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" event={"ID":"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f","Type":"ContainerStarted","Data":"a1c20745d789fe1915e4ff34a0f5d164e6f3f6408764f427a0d95d5ad410f135"} Oct 31 00:44:13.405551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:13.405489 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" podStartSLOduration=3.405472749 podStartE2EDuration="3.405472749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:13.402280134 +0000 UTC m=+1471.100393899" watchObservedRunningTime="2026-10-31 00:44:13.405472749 +0000 UTC m=+1471.103586511" Oct 31 00:44:13.816622 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:13.816584 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:13.995846 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:13.995787 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:14.373674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:14.373640 5978 generic.go:356] "Generic (PLEG): container finished" podID="c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f" containerID="beca1ccadc12bc772cd8ef0f29f587c0c587f9f482ce3f04c9061c0e3fcc8db8" exitCode=0 Oct 31 00:44:14.374187 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:14.373724 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" event={"ID":"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f","Type":"ContainerDied","Data":"beca1ccadc12bc772cd8ef0f29f587c0c587f9f482ce3f04c9061c0e3fcc8db8"} Oct 31 00:44:15.571873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.571845 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:15.653126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.653029 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-xks-ca\") pod \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " Oct 31 00:44:15.653126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.653101 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-openshift-ca\") pod \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " Oct 31 00:44:15.653363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.653324 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-kube-api-access-w8nhd\" (UniqueName: \"kubernetes.io/projected/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-kube-api-access-w8nhd\") pod \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\" (UID: \"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f\") " Oct 31 00:44:15.653547 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.653523 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-openshift-ca" pod "c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f" (UID: "c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:15.654706 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.654676 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:15.655948 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.655910 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-xks-ca" pod "c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f" (UID: "c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:15.656644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.656621 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-kube-api-access-w8nhd" pod "c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f" (UID: "c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f"). InnerVolumeSpecName "kube-api-access-w8nhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:15.756326 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.756288 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8nhd\" (UniqueName: \"kubernetes.io/projected/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-kube-api-access-w8nhd\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:15.756326 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:15.756326 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:16.392250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:16.392221 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" Oct 31 00:44:16.392403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:16.392255 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp" event={"ID":"c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f","Type":"ContainerDied","Data":"a1c20745d789fe1915e4ff34a0f5d164e6f3f6408764f427a0d95d5ad410f135"} Oct 31 00:44:16.392403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:16.392303 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1c20745d789fe1915e4ff34a0f5d164e6f3f6408764f427a0d95d5ad410f135" Oct 31 00:44:16.855273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:16.855236 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:18.531567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:18.531531 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:18.695741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:18.695709 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:20.626370 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:20.626334 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:20.782129 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:20.782091 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:21.594294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:21.594257 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:23.416131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:23.415968 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:23.688108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:23.688017 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:24.341409 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:24.341365 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:24.587736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:24.587695 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:25.518994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:25.518956 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:26.198313 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.198281 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd"] Oct 31 00:44:26.199177 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.199159 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f" containerName="revoke-keys" Oct 31 00:44:26.199221 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.199179 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f" containerName="revoke-keys" Oct 31 00:44:26.199302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.199293 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f" containerName="revoke-keys" Oct 31 00:44:26.204690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.204670 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.225731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.225687 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd"] Oct 31 00:44:26.228385 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.228360 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:26.228747 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.228732 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:44:26.364968 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.364934 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63505f52-5czqd\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.365182 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.365089 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6x6n\" (UniqueName: \"kubernetes.io/projected/9c8727fc-6db1-40c3-903a-0c61b1aaa810-kube-api-access-g6x6n\") pod \"maas-api-revoke-sub-63505f52-5czqd\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.365376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.365353 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63505f52-5czqd\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.467113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.467019 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63505f52-5czqd\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.467299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.467119 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63505f52-5czqd\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.467299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.467191 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6x6n\" (UniqueName: \"kubernetes.io/projected/9c8727fc-6db1-40c3-903a-0c61b1aaa810-kube-api-access-g6x6n\") pod \"maas-api-revoke-sub-63505f52-5czqd\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.467863 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.467838 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63505f52-5czqd\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.469557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.469532 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63505f52-5czqd\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.489074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.489036 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6x6n\" (UniqueName: \"kubernetes.io/projected/9c8727fc-6db1-40c3-903a-0c61b1aaa810-kube-api-access-g6x6n\") pod \"maas-api-revoke-sub-63505f52-5czqd\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.521826 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.521795 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:26.828302 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:44:26.828265 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8727fc_6db1_40c3_903a_0c61b1aaa810.slice/crio-02df2e75bd706e7a3109f7e27e5c0458e5618521e6051ccf510dfb05e45b72f0 WatchSource:0}: Error finding container 02df2e75bd706e7a3109f7e27e5c0458e5618521e6051ccf510dfb05e45b72f0: Status 404 returned error can't find the container with id 02df2e75bd706e7a3109f7e27e5c0458e5618521e6051ccf510dfb05e45b72f0 Oct 31 00:44:26.828808 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:26.828783 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd"] Oct 31 00:44:27.474430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:27.474336 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:27.505385 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:27.505337 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" event={"ID":"9c8727fc-6db1-40c3-903a-0c61b1aaa810","Type":"ContainerStarted","Data":"f798b374929cb589481730e91b03c954731ef65c896b35991f5b26d41e8364b4"} Oct 31 00:44:27.505586 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:27.505405 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" event={"ID":"9c8727fc-6db1-40c3-903a-0c61b1aaa810","Type":"ContainerStarted","Data":"02df2e75bd706e7a3109f7e27e5c0458e5618521e6051ccf510dfb05e45b72f0"} Oct 31 00:44:27.563906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:27.563851 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" podStartSLOduration=1.563836995 podStartE2EDuration="1.563836995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:27.559976401 +0000 UTC m=+1485.258090180" watchObservedRunningTime="2026-10-31 00:44:27.563836995 +0000 UTC m=+1485.261950760" Oct 31 00:44:27.677574 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:27.677524 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:27.843486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:27.839966 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:28.517395 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:28.517357 5978 generic.go:356] "Generic (PLEG): container finished" podID="9c8727fc-6db1-40c3-903a-0c61b1aaa810" containerID="f798b374929cb589481730e91b03c954731ef65c896b35991f5b26d41e8364b4" exitCode=0 Oct 31 00:44:28.518061 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:28.517445 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" event={"ID":"9c8727fc-6db1-40c3-903a-0c61b1aaa810","Type":"ContainerDied","Data":"f798b374929cb589481730e91b03c954731ef65c896b35991f5b26d41e8364b4"} Oct 31 00:44:28.710001 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:28.709966 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:28.990771 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:28.990729 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:29.609137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.609081 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:29.720177 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.720145 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:29.788204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.788096 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:29.862238 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.862198 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-xks-ca\") pod \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " Oct 31 00:44:29.862408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.862343 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-openshift-ca\") pod \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " Oct 31 00:44:29.862561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.862420 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8727fc-6db1-40c3-903a-0c61b1aaa810-kube-api-access-g6x6n\" (UniqueName: \"kubernetes.io/projected/9c8727fc-6db1-40c3-903a-0c61b1aaa810-kube-api-access-g6x6n\") pod \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\" (UID: \"9c8727fc-6db1-40c3-903a-0c61b1aaa810\") " Oct 31 00:44:29.864215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.864184 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-openshift-ca" pod "9c8727fc-6db1-40c3-903a-0c61b1aaa810" (UID: "9c8727fc-6db1-40c3-903a-0c61b1aaa810"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:29.866276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.866248 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-xks-ca" pod "9c8727fc-6db1-40c3-903a-0c61b1aaa810" (UID: "9c8727fc-6db1-40c3-903a-0c61b1aaa810"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:29.868731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.868695 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8727fc-6db1-40c3-903a-0c61b1aaa810-kube-api-access-g6x6n" pod "9c8727fc-6db1-40c3-903a-0c61b1aaa810" (UID: "9c8727fc-6db1-40c3-903a-0c61b1aaa810"). InnerVolumeSpecName "kube-api-access-g6x6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:29.965510 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.965476 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6x6n\" (UniqueName: \"kubernetes.io/projected/9c8727fc-6db1-40c3-903a-0c61b1aaa810-kube-api-access-g6x6n\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:29.965741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.965726 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:29.965842 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:29.965830 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8727fc-6db1-40c3-903a-0c61b1aaa810-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:30.541543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:30.541500 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" Oct 31 00:44:30.541732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:30.541502 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd" event={"ID":"9c8727fc-6db1-40c3-903a-0c61b1aaa810","Type":"ContainerDied","Data":"02df2e75bd706e7a3109f7e27e5c0458e5618521e6051ccf510dfb05e45b72f0"} Oct 31 00:44:30.541732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:30.541614 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02df2e75bd706e7a3109f7e27e5c0458e5618521e6051ccf510dfb05e45b72f0" Oct 31 00:44:30.728218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:30.728119 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:32.411840 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:32.411798 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:32.678282 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:32.678184 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:32.832237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:32.832197 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:33.190379 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:44:33.190341 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:44:33.378366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:33.378322 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:33.547791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:33.547707 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:34.700444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:34.700411 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:36.472213 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:36.472178 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:36.613344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:36.613304 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:36.763008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:36.762897 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:36.934991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:36.934955 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:37.090586 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:37.090550 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:37.261725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:37.257786 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:38.265064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:38.265022 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:38.422036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:38.422000 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:38.710715 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:38.710682 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:38.976636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:38.976541 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:39.934738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:39.934697 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:40.159939 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.159900 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:40.699878 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.699823 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g"] Oct 31 00:44:40.701374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.701346 5978 cpu_manager.go:414] "containerMap: removing container" podUID="9c8727fc-6db1-40c3-903a-0c61b1aaa810" containerName="revoke-keys" Oct 31 00:44:40.701559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.701543 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8727fc-6db1-40c3-903a-0c61b1aaa810" containerName="revoke-keys" Oct 31 00:44:40.701980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.701960 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8727fc-6db1-40c3-903a-0c61b1aaa810" containerName="revoke-keys" Oct 31 00:44:40.707052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.707027 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:40.723877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.723849 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:40.724123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.723950 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g"] Oct 31 00:44:40.731082 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.728591 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:44:40.905518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.903038 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b86619be-25f5g\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:40.905518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.903187 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b86619be-25f5g\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:40.905518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.903245 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrs8\" (UniqueName: \"kubernetes.io/projected/fad3a0c8-1c55-408e-9523-7ea822456671-kube-api-access-wmrs8\") pod \"maas-api-revoke-sub-b86619be-25f5g\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:40.939031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:40.938987 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:41.015993 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:41.015893 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b86619be-25f5g\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:41.016177 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:41.016058 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b86619be-25f5g\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:41.019769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:41.019730 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrs8\" (UniqueName: \"kubernetes.io/projected/fad3a0c8-1c55-408e-9523-7ea822456671-kube-api-access-wmrs8\") pod \"maas-api-revoke-sub-b86619be-25f5g\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:41.030006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:41.023643 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b86619be-25f5g\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:41.110167 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:41.110123 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b86619be-25f5g\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:41.192840 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:41.192275 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrs8\" (UniqueName: \"kubernetes.io/projected/fad3a0c8-1c55-408e-9523-7ea822456671-kube-api-access-wmrs8\") pod \"maas-api-revoke-sub-b86619be-25f5g\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:41.359980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:41.359661 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:42.895482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:42.875797 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.066717 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:43.066667 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g"] Oct 31 00:44:43.205590 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:43.133473 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:44:43.205590 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:43.190535 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:43.205590 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:43.190602 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.333492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:43.319562 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:44:43.356481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:43.333859 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:44:43.699509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:43.694621 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:43.797481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:43.722910 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.088543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.020283 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:44:44.160493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.141075 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 00:44:44.160493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.141542 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 00:44:44.247062 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.247005 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.368587 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.365709 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:44.465966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.465916 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:44.517884 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.517834 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:44:44.743018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.742964 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:44:45.101414 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.877747 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:45.102505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.102440 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.102704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.915481 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:44:45.102898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:44.905774 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.103054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.069881 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.204117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.204062 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.235880 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.235808 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 00:44:45.370546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.346482 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 31 00:44:45.614620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.371077 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.645955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.371141 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.957551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.371405 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.957551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.440769 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.957551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.489125 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.957551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.521590 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.957551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.738694 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" podUID="f22077e0-eac8-45ef-92fd-ac6f597563fa" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 00:44:45.957551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.761021 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.957551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.766051 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:45.957551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.822717 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:45.957551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:45.887893 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" podUID="c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 00:44:46.124559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.117565 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:46.142506 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:44:46.132028 5978 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.065s" Oct 31 00:44:46.142506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.132120 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:44:46.142506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.132173 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:44:46.142506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.132214 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:44:46.142506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.132343 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:44:46.142506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.132384 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:44:46.142506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.132423 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:44:46.142506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.132480 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:44:46.168524 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.145541 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:44:46.168524 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.145993 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:44:46.251851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.251642 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:46.329636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.329590 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:46.378720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.378681 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:46.379053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.378991 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:46.504533 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.501024 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:46.705507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.696174 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:46.826559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.792053 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 31 00:44:47.049988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.847773 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:44:47.050272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.050240 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.050404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.935660 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.050867 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.921099 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.051181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.935698 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:44:47.051552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.935720 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:44:47.051894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:46.963824 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.221570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.167055 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.624630 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.624592 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:47.624983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.178837 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:44:47.656634 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.269206 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.658608 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.352303 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.718481 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:44:47.715006 5978 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.582s" Oct 31 00:44:47.887920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.764169 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Oct 31 00:44:47.887920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.777586 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:44:47.887920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.837755 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-72ff4" podUID="a3a55405-5cba-4424-b346-9b2c8d98b3a0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:47.887920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:47.178803 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:44:48.037697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.037653 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:48.400610 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.351589 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:44:48.400610 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.351927 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:48.400610 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.352140 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:44:48.400610 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.352315 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:44:48.693191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.692988 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.730561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.693382 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:44:48.730561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.693403 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 00:44:48.730561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.693342 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:48.820528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.806182 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:48.865692 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.865646 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:44:48.901502 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:48.899547 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.039872 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:44:49.039778 5978 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.321s" Oct 31 00:44:49.061950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.061912 5978 generic.go:356] "Generic (PLEG): container finished" podID="fad3a0c8-1c55-408e-9523-7ea822456671" containerID="c067103e2554a4769cbf8c82be6e1d836737e7de93ca1349fd813965113a97bb" exitCode=0 Oct 31 00:44:49.367474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.361540 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.376479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.376407 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:44:49.377219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.377177 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:44:49.377742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.377720 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:44:49.523032 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.522992 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:44:49.523582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.523085 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:44:49.523582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.523159 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:44:49.523582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.523180 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" event={"ID":"fad3a0c8-1c55-408e-9523-7ea822456671","Type":"ContainerStarted","Data":"8a86c522aa09c1785146352e89aa29a4cca903bec1804b0097092d38b40b598d"} Oct 31 00:44:49.523582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.523228 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" event={"ID":"fad3a0c8-1c55-408e-9523-7ea822456671","Type":"ContainerDied","Data":"c067103e2554a4769cbf8c82be6e1d836737e7de93ca1349fd813965113a97bb"} Oct 31 00:44:49.667539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.666008 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:44:49.713063 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.713022 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 00:44:49.714293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.714259 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:44:49.714838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.714809 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:44:49.802881 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.802845 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:44:49.876588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.876287 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:44:49.882155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.880932 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 00:44:49.882155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.880976 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:44:49.882155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.881192 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:44:49.898789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.883504 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:44:49.924588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.923719 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 00:44:49.926036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.926012 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 00:44:49.940174 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.940145 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:44:49.952155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:49.952130 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 00:44:50.090013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.089982 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:44:50.094714 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.094688 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:44:50.369009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.368978 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:50.406778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.406739 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:50.530483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.530385 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-xks-ca\") pod \"fad3a0c8-1c55-408e-9523-7ea822456671\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " Oct 31 00:44:50.530921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.530535 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-openshift-ca\") pod \"fad3a0c8-1c55-408e-9523-7ea822456671\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " Oct 31 00:44:50.530921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.530837 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad3a0c8-1c55-408e-9523-7ea822456671-kube-api-access-wmrs8\" (UniqueName: \"kubernetes.io/projected/fad3a0c8-1c55-408e-9523-7ea822456671-kube-api-access-wmrs8\") pod \"fad3a0c8-1c55-408e-9523-7ea822456671\" (UID: \"fad3a0c8-1c55-408e-9523-7ea822456671\") " Oct 31 00:44:50.530921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.530858 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-openshift-ca" pod "fad3a0c8-1c55-408e-9523-7ea822456671" (UID: "fad3a0c8-1c55-408e-9523-7ea822456671"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:50.531536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.531512 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:50.532761 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.532728 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-xks-ca" pod "fad3a0c8-1c55-408e-9523-7ea822456671" (UID: "fad3a0c8-1c55-408e-9523-7ea822456671"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:50.533099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.533075 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad3a0c8-1c55-408e-9523-7ea822456671-kube-api-access-wmrs8" pod "fad3a0c8-1c55-408e-9523-7ea822456671" (UID: "fad3a0c8-1c55-408e-9523-7ea822456671"). InnerVolumeSpecName "kube-api-access-wmrs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:50.562848 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.562810 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:50.635727 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.634141 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrs8\" (UniqueName: \"kubernetes.io/projected/fad3a0c8-1c55-408e-9523-7ea822456671-kube-api-access-wmrs8\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:50.635727 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.634180 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad3a0c8-1c55-408e-9523-7ea822456671-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:44:50.795096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.794991 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658"] Oct 31 00:44:50.801656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.796239 5978 cpu_manager.go:414] "containerMap: removing container" podUID="fad3a0c8-1c55-408e-9523-7ea822456671" containerName="revoke-keys" Oct 31 00:44:50.801656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.796267 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad3a0c8-1c55-408e-9523-7ea822456671" containerName="revoke-keys" Oct 31 00:44:50.801656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.796502 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad3a0c8-1c55-408e-9523-7ea822456671" containerName="revoke-keys" Oct 31 00:44:50.835765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.835726 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658"] Oct 31 00:44:50.836161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.836135 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:44:50.852642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.851843 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b08f569a\"" Oct 31 00:44:50.852642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.852098 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b08f569a-dockercfg-nlzbg\"" Oct 31 00:44:50.942114 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.942076 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8785f3-8a95-4ac7-8512-5df4123f2fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658\" (UID: \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:44:50.942311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.942140 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5fk\" (UniqueName: \"kubernetes.io/projected/8c8785f3-8a95-4ac7-8512-5df4123f2fda-kube-api-access-vf5fk\") pod \"payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658\" (UID: \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:44:50.967816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.967780 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp"] Oct 31 00:44:50.972734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.972709 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:44:50.996618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:50.996581 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp"] Oct 31 00:44:51.043941 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.043824 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8785f3-8a95-4ac7-8512-5df4123f2fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658\" (UID: \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:44:51.044247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.044216 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5fk\" (UniqueName: \"kubernetes.io/projected/8c8785f3-8a95-4ac7-8512-5df4123f2fda-kube-api-access-vf5fk\") pod \"payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658\" (UID: \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:44:51.044877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.044851 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8785f3-8a95-4ac7-8512-5df4123f2fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658\" (UID: \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:44:51.079144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.079110 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5fk\" (UniqueName: \"kubernetes.io/projected/8c8785f3-8a95-4ac7-8512-5df4123f2fda-kube-api-access-vf5fk\") pod \"payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658\" (UID: \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:44:51.132251 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.132218 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" Oct 31 00:44:51.132471 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.132245 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g" event={"ID":"fad3a0c8-1c55-408e-9523-7ea822456671","Type":"ContainerDied","Data":"8a86c522aa09c1785146352e89aa29a4cca903bec1804b0097092d38b40b598d"} Oct 31 00:44:51.132471 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.132322 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a86c522aa09c1785146352e89aa29a4cca903bec1804b0097092d38b40b598d" Oct 31 00:44:51.146407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.146375 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpwk\" (UniqueName: \"kubernetes.io/projected/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-kube-api-access-nzpwk\") pod \"payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp\" (UID: \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\") " pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:44:51.146608 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.146587 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-plugins-config-volume\") pod \"payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp\" (UID: \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\") " pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:44:51.178894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.178857 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:44:51.257125 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.256672 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-plugins-config-volume\") pod \"payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp\" (UID: \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\") " pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:44:51.257125 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.257053 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpwk\" (UniqueName: \"kubernetes.io/projected/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-kube-api-access-nzpwk\") pod \"payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp\" (UID: \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\") " pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:44:51.258796 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.258768 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-plugins-config-volume\") pod \"payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp\" (UID: \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\") " pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:44:51.317663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.315069 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpwk\" (UniqueName: \"kubernetes.io/projected/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-kube-api-access-nzpwk\") pod \"payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp\" (UID: \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\") " pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:44:51.518222 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.518184 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658"] Oct 31 00:44:51.520347 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:44:51.520317 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8785f3_8a95_4ac7_8512_5df4123f2fda.slice/crio-ea4402e7375b97bbff6475be8bdd1d8bfa7c5d71dc687812b7ff84dae7fc0c28 WatchSource:0}: Error finding container ea4402e7375b97bbff6475be8bdd1d8bfa7c5d71dc687812b7ff84dae7fc0c28: Status 404 returned error can't find the container with id ea4402e7375b97bbff6475be8bdd1d8bfa7c5d71dc687812b7ff84dae7fc0c28 Oct 31 00:44:51.594288 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.594252 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:44:51.981159 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:51.981133 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp"] Oct 31 00:44:51.981308 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:44:51.981285 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0534b4c0_79f4_47bd_9ff5_ec6b62da4777.slice/crio-a361517c8eec0a06039ca10872252fb9e180fb4e43a7c9834b0f335573defa11 WatchSource:0}: Error finding container a361517c8eec0a06039ca10872252fb9e180fb4e43a7c9834b0f335573defa11: Status 404 returned error can't find the container with id a361517c8eec0a06039ca10872252fb9e180fb4e43a7c9834b0f335573defa11 Oct 31 00:44:52.145728 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:52.145691 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" event={"ID":"0534b4c0-79f4-47bd-9ff5-ec6b62da4777","Type":"ContainerStarted","Data":"e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae"} Oct 31 00:44:52.145937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:52.145756 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" event={"ID":"0534b4c0-79f4-47bd-9ff5-ec6b62da4777","Type":"ContainerStarted","Data":"a361517c8eec0a06039ca10872252fb9e180fb4e43a7c9834b0f335573defa11"} Oct 31 00:44:52.145937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:52.145809 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:44:52.147709 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:52.147680 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" event={"ID":"8c8785f3-8a95-4ac7-8512-5df4123f2fda","Type":"ContainerStarted","Data":"84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e"} Oct 31 00:44:52.147849 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:52.147725 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" event={"ID":"8c8785f3-8a95-4ac7-8512-5df4123f2fda","Type":"ContainerStarted","Data":"ea4402e7375b97bbff6475be8bdd1d8bfa7c5d71dc687812b7ff84dae7fc0c28"} Oct 31 00:44:52.147849 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:52.147786 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:44:52.187551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:52.187495 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" podStartSLOduration=2.187481236 podStartE2EDuration="2.187481236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:52.182362163 +0000 UTC m=+1509.880475921" watchObservedRunningTime="2026-10-31 00:44:52.187481236 +0000 UTC m=+1509.885594998" Oct 31 00:44:52.222292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:52.222231 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" podStartSLOduration=2.222216792 podStartE2EDuration="2.222216792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:52.218206003 +0000 UTC m=+1509.916319768" watchObservedRunningTime="2026-10-31 00:44:52.222216792 +0000 UTC m=+1509.920330582" Oct 31 00:44:53.143380 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:53.143348 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:54.807942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:54.807904 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:54.974304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:54.974253 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:55.175737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:55.175696 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:56.732866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:56.732824 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:58.538569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:44:58.538476 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.131925 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.131879 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f"] Oct 31 00:45:00.137479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.137380 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.155983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.155942 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f598cd05-serving-cert\"" Oct 31 00:45:00.174017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.173981 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f"] Oct 31 00:45:00.201760 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.201707 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv"] Oct 31 00:45:00.222339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.222308 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj"] Oct 31 00:45:00.224577 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.222269 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:00.230917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.230566 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.230917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.230613 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgjb\" (UniqueName: \"kubernetes.io/projected/6fe078f0-9060-431d-8fe4-59ec4436c5d8-kube-api-access-ghgjb\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.230917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.230696 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe078f0-9060-431d-8fe4-59ec4436c5d8-openshift-service-ca\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.230917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.230782 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-metrics-certs\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.231865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.231430 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4"] Oct 31 00:45:00.232292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.232126 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:00.241723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.239967 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:00.241723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.241624 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv"] Oct 31 00:45:00.249985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.249954 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj"] Oct 31 00:45:00.257089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.257054 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4"] Oct 31 00:45:00.264238 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.264207 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd"] Oct 31 00:45:00.264720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.264694 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:45:00.277266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.275209 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk"] Oct 31 00:45:00.277266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.275345 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:00.282413 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.282313 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:00.291913 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.291882 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk"] Oct 31 00:45:00.295239 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.295193 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.298566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.298540 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd"] Oct 31 00:45:00.332134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.332096 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.332134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.332135 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgjb\" (UniqueName: \"kubernetes.io/projected/6fe078f0-9060-431d-8fe4-59ec4436c5d8-kube-api-access-ghgjb\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.332415 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:00.332209 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-serving-cert: secret "maas-api-e2e-migrate-f598cd05-serving-cert" not found Oct 31 00:45:00.332415 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.332247 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4\" (UID: \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:00.332415 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:00.332280 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls podName:6fe078f0-9060-431d-8fe4-59ec4436c5d8 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:00.832259565 +0000 UTC m=+1518.530373330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls") pod "maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" (UID: "6fe078f0-9060-431d-8fe4-59ec4436c5d8") : secret "maas-api-e2e-migrate-f598cd05-serving-cert" not found Oct 31 00:45:00.332415 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.332373 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbgfw\" (UniqueName: \"kubernetes.io/projected/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-kube-api-access-gbgfw\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4\" (UID: \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:00.332415 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.332415 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe078f0-9060-431d-8fe4-59ec4436c5d8-openshift-service-ca\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.332659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.332618 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-metrics-certs\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.332750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.332731 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmlhk\" (UniqueName: \"kubernetes.io/projected/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-kube-api-access-wmlhk\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv\" (UID: \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:00.332871 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.332852 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv\" (UID: \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:00.333277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.333052 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-mw9vj\" (UID: \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:00.333277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.333097 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trvgv\" (UniqueName: \"kubernetes.io/projected/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-kube-api-access-trvgv\") pod \"maas-api-key-cleanup-29890125-mw9vj\" (UID: \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:00.333277 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.333218 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe078f0-9060-431d-8fe4-59ec4436c5d8-openshift-service-ca\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.335487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.335445 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-metrics-certs\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.389756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.389625 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgjb\" (UniqueName: \"kubernetes.io/projected/6fe078f0-9060-431d-8fe4-59ec4436c5d8-kube-api-access-ghgjb\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.434595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.434553 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv\" (UID: \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:00.434824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.434715 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/dd009a89-07a6-4b32-bf9e-9969715eb647-kube-api-access-9nqkz\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd\" (UID: \"dd009a89-07a6-4b32-bf9e-9969715eb647\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:00.434895 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.434852 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-mw9vj\" (UID: \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:00.434895 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.434891 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvgv\" (UniqueName: \"kubernetes.io/projected/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-kube-api-access-trvgv\") pod \"maas-api-key-cleanup-29890125-mw9vj\" (UID: \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:00.435020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.434933 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcba1e-3411-466d-ab12-965d3d34ad44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk\" (UID: \"c8fcba1e-3411-466d-ab12-965d3d34ad44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:00.435020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.434958 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nqct\" (UniqueName: \"kubernetes.io/projected/c8fcba1e-3411-466d-ab12-965d3d34ad44-kube-api-access-9nqct\") pod \"maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk\" (UID: \"c8fcba1e-3411-466d-ab12-965d3d34ad44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:00.435020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.435000 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4\" (UID: \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:00.435202 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.435065 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd009a89-07a6-4b32-bf9e-9969715eb647-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd\" (UID: \"dd009a89-07a6-4b32-bf9e-9969715eb647\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:00.435275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.435205 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbgfw\" (UniqueName: \"kubernetes.io/projected/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-kube-api-access-gbgfw\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4\" (UID: \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:00.435412 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.435386 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmlhk\" (UniqueName: \"kubernetes.io/projected/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-kube-api-access-wmlhk\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv\" (UID: \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:00.435538 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.435406 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv\" (UID: \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:00.435710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.435688 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-mw9vj\" (UID: \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:00.435796 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.435775 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4\" (UID: \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:00.442440 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.442405 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.456973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.456933 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmlhk\" (UniqueName: \"kubernetes.io/projected/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-kube-api-access-wmlhk\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv\" (UID: \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:00.457139 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.456981 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvgv\" (UniqueName: \"kubernetes.io/projected/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-kube-api-access-trvgv\") pod \"maas-api-key-cleanup-29890125-mw9vj\" (UID: \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:00.457139 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.457101 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbgfw\" (UniqueName: \"kubernetes.io/projected/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-kube-api-access-gbgfw\") pod \"maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4\" (UID: \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:00.537075 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.537026 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcba1e-3411-466d-ab12-965d3d34ad44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk\" (UID: \"c8fcba1e-3411-466d-ab12-965d3d34ad44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:00.537294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.537086 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqct\" (UniqueName: \"kubernetes.io/projected/c8fcba1e-3411-466d-ab12-965d3d34ad44-kube-api-access-9nqct\") pod \"maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk\" (UID: \"c8fcba1e-3411-466d-ab12-965d3d34ad44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:00.537294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.537161 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd009a89-07a6-4b32-bf9e-9969715eb647-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd\" (UID: \"dd009a89-07a6-4b32-bf9e-9969715eb647\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:00.537534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.537514 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/dd009a89-07a6-4b32-bf9e-9969715eb647-kube-api-access-9nqkz\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd\" (UID: \"dd009a89-07a6-4b32-bf9e-9969715eb647\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:00.538031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.538001 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcba1e-3411-466d-ab12-965d3d34ad44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk\" (UID: \"c8fcba1e-3411-466d-ab12-965d3d34ad44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:00.538134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.538009 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd009a89-07a6-4b32-bf9e-9969715eb647-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd\" (UID: \"dd009a89-07a6-4b32-bf9e-9969715eb647\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:00.556365 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.556331 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:00.558678 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.558649 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqct\" (UniqueName: \"kubernetes.io/projected/c8fcba1e-3411-466d-ab12-965d3d34ad44-kube-api-access-9nqct\") pod \"maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk\" (UID: \"c8fcba1e-3411-466d-ab12-965d3d34ad44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:00.558823 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.558686 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/dd009a89-07a6-4b32-bf9e-9969715eb647-kube-api-access-9nqkz\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd\" (UID: \"dd009a89-07a6-4b32-bf9e-9969715eb647\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:00.587576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.587545 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:00.593292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.593260 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.606546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.606504 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:00.611622 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.610635 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:00.621339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.621300 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:00.754810 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.754771 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.846000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.845957 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.855195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.855164 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls\") pod \"maas-api-e2e-migrate-f598cd05-944f49478-ptb2f\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:00.872894 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:45:00.872847 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b946e9a_4b0f_49c7_ac5b_cb39368f56eb.slice/crio-016cf78196d5ad4edfbc9902b4ee0ce4f3f9a4695a49065609f6645cbb0612fe WatchSource:0}: Error finding container 016cf78196d5ad4edfbc9902b4ee0ce4f3f9a4695a49065609f6645cbb0612fe: Status 404 returned error can't find the container with id 016cf78196d5ad4edfbc9902b4ee0ce4f3f9a4695a49065609f6645cbb0612fe Oct 31 00:45:00.902480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.900683 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv"] Oct 31 00:45:00.933547 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.933519 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj"] Oct 31 00:45:00.935900 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:45:00.935874 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a243edd_e5c8_4bdf_8174_9a9f5542f5d9.slice/crio-55f4d51300aa7cbda9b3c84588a5343f246b21213d4c940aa62b7d432145e173 WatchSource:0}: Error finding container 55f4d51300aa7cbda9b3c84588a5343f246b21213d4c940aa62b7d432145e173: Status 404 returned error can't find the container with id 55f4d51300aa7cbda9b3c84588a5343f246b21213d4c940aa62b7d432145e173 Oct 31 00:45:00.983877 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:45:00.983832 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916c10a6_5fe3_4d5f_872e_ab58abaca3eb.slice/crio-eb2eed21d223e417d91af778bbaf1ec656a1777fde6a25034ac0207b6cebed62 WatchSource:0}: Error finding container eb2eed21d223e417d91af778bbaf1ec656a1777fde6a25034ac0207b6cebed62: Status 404 returned error can't find the container with id eb2eed21d223e417d91af778bbaf1ec656a1777fde6a25034ac0207b6cebed62 Oct 31 00:45:00.984835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:00.984807 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4"] Oct 31 00:45:01.012733 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:45:01.012696 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd009a89_07a6_4b32_bf9e_9969715eb647.slice/crio-4f3069c4652a20d3e58dc5991f319022a47f46369828abad29f54a5e968bf97e WatchSource:0}: Error finding container 4f3069c4652a20d3e58dc5991f319022a47f46369828abad29f54a5e968bf97e: Status 404 returned error can't find the container with id 4f3069c4652a20d3e58dc5991f319022a47f46369828abad29f54a5e968bf97e Oct 31 00:45:01.015117 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.015085 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd"] Oct 31 00:45:01.054224 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.054190 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:01.081982 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.081942 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk"] Oct 31 00:45:01.116023 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:45:01.115978 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fcba1e_3411_466d_ab12_965d3d34ad44.slice/crio-fb5cdc9c70f13cca9457c505c72d2eec7961998a3366a8211eff4b52051445d0 WatchSource:0}: Error finding container fb5cdc9c70f13cca9457c505c72d2eec7961998a3366a8211eff4b52051445d0: Status 404 returned error can't find the container with id fb5cdc9c70f13cca9457c505c72d2eec7961998a3366a8211eff4b52051445d0 Oct 31 00:45:01.270559 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.270521 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" event={"ID":"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb","Type":"ContainerStarted","Data":"016cf78196d5ad4edfbc9902b4ee0ce4f3f9a4695a49065609f6645cbb0612fe"} Oct 31 00:45:01.272350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.272316 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" event={"ID":"916c10a6-5fe3-4d5f-872e-ab58abaca3eb","Type":"ContainerStarted","Data":"eb2eed21d223e417d91af778bbaf1ec656a1777fde6a25034ac0207b6cebed62"} Oct 31 00:45:01.274429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.274401 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" event={"ID":"c8fcba1e-3411-466d-ab12-965d3d34ad44","Type":"ContainerStarted","Data":"fb5cdc9c70f13cca9457c505c72d2eec7961998a3366a8211eff4b52051445d0"} Oct 31 00:45:01.276540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.276509 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" event={"ID":"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9","Type":"ContainerStarted","Data":"55f4d51300aa7cbda9b3c84588a5343f246b21213d4c940aa62b7d432145e173"} Oct 31 00:45:01.289115 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.289008 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" event={"ID":"dd009a89-07a6-4b32-bf9e-9969715eb647","Type":"ContainerStarted","Data":"4f3069c4652a20d3e58dc5991f319022a47f46369828abad29f54a5e968bf97e"} Oct 31 00:45:01.357173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.357139 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:01.565553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:01.560017 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f"] Oct 31 00:45:01.565553 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:45:01.560518 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe078f0_9060_431d_8fe4_59ec4436c5d8.slice/crio-e29cf160da46e86568f55848a5595c9c96d795ca43c336b61baea39ff91f96e6 WatchSource:0}: Error finding container e29cf160da46e86568f55848a5595c9c96d795ca43c336b61baea39ff91f96e6: Status 404 returned error can't find the container with id e29cf160da46e86568f55848a5595c9c96d795ca43c336b61baea39ff91f96e6 Oct 31 00:45:02.191731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.191698 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:02.310779 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.310738 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" event={"ID":"6fe078f0-9060-431d-8fe4-59ec4436c5d8","Type":"ContainerStarted","Data":"516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2"} Oct 31 00:45:02.311266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.310801 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" event={"ID":"6fe078f0-9060-431d-8fe4-59ec4436c5d8","Type":"ContainerStarted","Data":"e29cf160da46e86568f55848a5595c9c96d795ca43c336b61baea39ff91f96e6"} Oct 31 00:45:02.311266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.310906 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:02.313029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.312997 5978 generic.go:356] "Generic (PLEG): container finished" podID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerID="9a8c096442ab20a0e8f954c05499abfa60ead7ec00a5430f0cbf2b0592451224" exitCode=60 Oct 31 00:45:02.313509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.313079 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" event={"ID":"916c10a6-5fe3-4d5f-872e-ab58abaca3eb","Type":"ContainerDied","Data":"9a8c096442ab20a0e8f954c05499abfa60ead7ec00a5430f0cbf2b0592451224"} Oct 31 00:45:02.313663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.313591 5978 scope.go:122] "RemoveContainer" containerID="9a8c096442ab20a0e8f954c05499abfa60ead7ec00a5430f0cbf2b0592451224" Oct 31 00:45:02.316718 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.315412 5978 generic.go:356] "Generic (PLEG): container finished" podID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerID="034f1c0521ead29b44bd24e29b4c06b3d81e133e3afe2309e0a1444016b1eb6e" exitCode=7 Oct 31 00:45:02.316718 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.315531 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" event={"ID":"c8fcba1e-3411-466d-ab12-965d3d34ad44","Type":"ContainerDied","Data":"034f1c0521ead29b44bd24e29b4c06b3d81e133e3afe2309e0a1444016b1eb6e"} Oct 31 00:45:02.316718 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.316083 5978 scope.go:122] "RemoveContainer" containerID="034f1c0521ead29b44bd24e29b4c06b3d81e133e3afe2309e0a1444016b1eb6e" Oct 31 00:45:02.324813 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.324780 5978 generic.go:356] "Generic (PLEG): container finished" podID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerID="604ef7c858cf405ce21f034170b0a38498d168cb0afff8d7a8a8e3fdb0fc2814" exitCode=60 Oct 31 00:45:02.325111 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.325083 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" event={"ID":"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9","Type":"ContainerDied","Data":"604ef7c858cf405ce21f034170b0a38498d168cb0afff8d7a8a8e3fdb0fc2814"} Oct 31 00:45:02.325509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.325442 5978 scope.go:122] "RemoveContainer" containerID="604ef7c858cf405ce21f034170b0a38498d168cb0afff8d7a8a8e3fdb0fc2814" Oct 31 00:45:02.328127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.328098 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" event={"ID":"dd009a89-07a6-4b32-bf9e-9969715eb647","Type":"ContainerStarted","Data":"3e7f5fc1a4aa420bf7835fd8e3db3cdb15648819f65a7c803608d78961f09472"} Oct 31 00:45:02.331795 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.331767 5978 generic.go:356] "Generic (PLEG): container finished" podID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerID="9f6dbdf14b29f80e1143f76d77e30fac21740250cf78a0701bde5d53c3d982a9" exitCode=60 Oct 31 00:45:02.331938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.331854 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" event={"ID":"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb","Type":"ContainerDied","Data":"9f6dbdf14b29f80e1143f76d77e30fac21740250cf78a0701bde5d53c3d982a9"} Oct 31 00:45:02.332209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.332184 5978 scope.go:122] "RemoveContainer" containerID="9f6dbdf14b29f80e1143f76d77e30fac21740250cf78a0701bde5d53c3d982a9" Oct 31 00:45:02.350824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.350742 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" podStartSLOduration=1.879380252 podStartE2EDuration="2.350716986s" totalImagesPullingTime="471.336734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:02.349023921 +0000 UTC m=+1520.047137709" watchObservedRunningTime="2026-10-31 00:45:02.350716986 +0000 UTC m=+1520.048830756" Oct 31 00:45:02.546561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:02.546398 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" podStartSLOduration=2.546346178 podStartE2EDuration="2.546346178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:02.544744335 +0000 UTC m=+1520.242858100" watchObservedRunningTime="2026-10-31 00:45:02.546346178 +0000 UTC m=+1520.244459943" Oct 31 00:45:03.163665 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.163577 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:45:03.163665 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.163653 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:45:03.186318 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:03.186274 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:45:03.343048 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.343008 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" event={"ID":"916c10a6-5fe3-4d5f-872e-ab58abaca3eb","Type":"ContainerStarted","Data":"ce814c8ba5548d4c54da36770bb399d7f2177fe1f9d2c29c3d7132a4fa67cee6"} Oct 31 00:45:03.345244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.345214 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" event={"ID":"c8fcba1e-3411-466d-ab12-965d3d34ad44","Type":"ContainerStarted","Data":"119f5ca7c535afd00771b1a44bc7313f60fd03fd5a42496e8c7fd45445453534"} Oct 31 00:45:03.347330 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.347300 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" event={"ID":"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9","Type":"ContainerStarted","Data":"a0371b6d73becdb99258a11c405685b81527783bc756053c9f645d137d10e4a5"} Oct 31 00:45:03.348952 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.348929 5978 generic.go:356] "Generic (PLEG): container finished" podID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerID="3e7f5fc1a4aa420bf7835fd8e3db3cdb15648819f65a7c803608d78961f09472" exitCode=60 Oct 31 00:45:03.349071 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.348977 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" event={"ID":"dd009a89-07a6-4b32-bf9e-9969715eb647","Type":"ContainerDied","Data":"3e7f5fc1a4aa420bf7835fd8e3db3cdb15648819f65a7c803608d78961f09472"} Oct 31 00:45:03.349305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.349288 5978 scope.go:122] "RemoveContainer" containerID="3e7f5fc1a4aa420bf7835fd8e3db3cdb15648819f65a7c803608d78961f09472" Oct 31 00:45:03.351266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.351146 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" event={"ID":"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb","Type":"ContainerStarted","Data":"3f3ecd7394815714bf788854ce9db958869a70f74523d832926720020e466b66"} Oct 31 00:45:03.399123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.399063 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" podStartSLOduration=3.39904852 podStartE2EDuration="3.39904852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.395502772 +0000 UTC m=+1521.093616537" watchObservedRunningTime="2026-10-31 00:45:03.39904852 +0000 UTC m=+1521.097162285" Oct 31 00:45:03.493286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.493174 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" podStartSLOduration=3.493155215 podStartE2EDuration="3.493155215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.490393392 +0000 UTC m=+1521.188507157" watchObservedRunningTime="2026-10-31 00:45:03.493155215 +0000 UTC m=+1521.191268980" Oct 31 00:45:03.533622 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.533573 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" podStartSLOduration=3.533556676 podStartE2EDuration="3.533556676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.530448395 +0000 UTC m=+1521.228562160" watchObservedRunningTime="2026-10-31 00:45:03.533556676 +0000 UTC m=+1521.231670464" Oct 31 00:45:03.587484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:03.587412 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" podStartSLOduration=3.587393955 podStartE2EDuration="3.587393955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.584052322 +0000 UTC m=+1521.282166086" watchObservedRunningTime="2026-10-31 00:45:03.587393955 +0000 UTC m=+1521.285507752" Oct 31 00:45:04.046985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.046892 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:04.364759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.364725 5978 generic.go:356] "Generic (PLEG): container finished" podID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerID="ce814c8ba5548d4c54da36770bb399d7f2177fe1f9d2c29c3d7132a4fa67cee6" exitCode=60 Oct 31 00:45:04.365268 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.364810 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" event={"ID":"916c10a6-5fe3-4d5f-872e-ab58abaca3eb","Type":"ContainerDied","Data":"ce814c8ba5548d4c54da36770bb399d7f2177fe1f9d2c29c3d7132a4fa67cee6"} Oct 31 00:45:04.365268 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.364876 5978 scope.go:122] "RemoveContainer" containerID="9a8c096442ab20a0e8f954c05499abfa60ead7ec00a5430f0cbf2b0592451224" Oct 31 00:45:04.365268 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.365173 5978 scope.go:122] "RemoveContainer" containerID="ce814c8ba5548d4c54da36770bb399d7f2177fe1f9d2c29c3d7132a4fa67cee6" Oct 31 00:45:04.365726 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:04.365701 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4_odh-ai-gateway-infra(916c10a6-5fe3-4d5f-872e-ab58abaca3eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" Oct 31 00:45:04.367321 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.367295 5978 generic.go:356] "Generic (PLEG): container finished" podID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerID="119f5ca7c535afd00771b1a44bc7313f60fd03fd5a42496e8c7fd45445453534" exitCode=60 Oct 31 00:45:04.367427 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.367365 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" event={"ID":"c8fcba1e-3411-466d-ab12-965d3d34ad44","Type":"ContainerDied","Data":"119f5ca7c535afd00771b1a44bc7313f60fd03fd5a42496e8c7fd45445453534"} Oct 31 00:45:04.367827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.367808 5978 scope.go:122] "RemoveContainer" containerID="119f5ca7c535afd00771b1a44bc7313f60fd03fd5a42496e8c7fd45445453534" Oct 31 00:45:04.368276 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:04.368252 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk_odh-ai-gateway-infra(c8fcba1e-3411-466d-ab12-965d3d34ad44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" Oct 31 00:45:04.370251 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.370222 5978 generic.go:356] "Generic (PLEG): container finished" podID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerID="a0371b6d73becdb99258a11c405685b81527783bc756053c9f645d137d10e4a5" exitCode=60 Oct 31 00:45:04.370389 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.370364 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" event={"ID":"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9","Type":"ContainerDied","Data":"a0371b6d73becdb99258a11c405685b81527783bc756053c9f645d137d10e4a5"} Oct 31 00:45:04.370806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.370785 5978 scope.go:122] "RemoveContainer" containerID="a0371b6d73becdb99258a11c405685b81527783bc756053c9f645d137d10e4a5" Oct 31 00:45:04.371260 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:04.371232 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-mw9vj_odh-ai-gateway-infra(4a243edd-e5c8-4bdf-8174-9a9f5542f5d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" Oct 31 00:45:04.373379 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.373349 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" event={"ID":"dd009a89-07a6-4b32-bf9e-9969715eb647","Type":"ContainerStarted","Data":"cd034b6e6c91b00ee885aa43469f5df1ba638840b7ace9089be5290f8f973d98"} Oct 31 00:45:04.375527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.375503 5978 generic.go:356] "Generic (PLEG): container finished" podID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerID="3f3ecd7394815714bf788854ce9db958869a70f74523d832926720020e466b66" exitCode=60 Oct 31 00:45:04.375636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.375550 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" event={"ID":"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb","Type":"ContainerDied","Data":"3f3ecd7394815714bf788854ce9db958869a70f74523d832926720020e466b66"} Oct 31 00:45:04.375947 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.375926 5978 scope.go:122] "RemoveContainer" containerID="3f3ecd7394815714bf788854ce9db958869a70f74523d832926720020e466b66" Oct 31 00:45:04.376310 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:04.376285 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv_odh-ai-gateway-infra(1b946e9a-4b0f-49c7-ac5b-cb39368f56eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" Oct 31 00:45:04.381178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.381154 5978 scope.go:122] "RemoveContainer" containerID="034f1c0521ead29b44bd24e29b4c06b3d81e133e3afe2309e0a1444016b1eb6e" Oct 31 00:45:04.396613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.396590 5978 scope.go:122] "RemoveContainer" containerID="604ef7c858cf405ce21f034170b0a38498d168cb0afff8d7a8a8e3fdb0fc2814" Oct 31 00:45:04.414007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:04.413981 5978 scope.go:122] "RemoveContainer" containerID="9f6dbdf14b29f80e1143f76d77e30fac21740250cf78a0701bde5d53c3d982a9" Oct 31 00:45:05.386949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:05.386914 5978 scope.go:122] "RemoveContainer" containerID="ce814c8ba5548d4c54da36770bb399d7f2177fe1f9d2c29c3d7132a4fa67cee6" Oct 31 00:45:05.387485 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:05.387426 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4_odh-ai-gateway-infra(916c10a6-5fe3-4d5f-872e-ab58abaca3eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" Oct 31 00:45:05.388754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:05.388732 5978 scope.go:122] "RemoveContainer" containerID="119f5ca7c535afd00771b1a44bc7313f60fd03fd5a42496e8c7fd45445453534" Oct 31 00:45:05.389156 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:05.389134 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk_odh-ai-gateway-infra(c8fcba1e-3411-466d-ab12-965d3d34ad44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" Oct 31 00:45:05.390400 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:05.390381 5978 scope.go:122] "RemoveContainer" containerID="a0371b6d73becdb99258a11c405685b81527783bc756053c9f645d137d10e4a5" Oct 31 00:45:05.390801 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:05.390779 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-mw9vj_odh-ai-gateway-infra(4a243edd-e5c8-4bdf-8174-9a9f5542f5d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" Oct 31 00:45:05.392152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:05.392128 5978 generic.go:356] "Generic (PLEG): container finished" podID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerID="cd034b6e6c91b00ee885aa43469f5df1ba638840b7ace9089be5290f8f973d98" exitCode=60 Oct 31 00:45:05.392259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:05.392207 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" event={"ID":"dd009a89-07a6-4b32-bf9e-9969715eb647","Type":"ContainerDied","Data":"cd034b6e6c91b00ee885aa43469f5df1ba638840b7ace9089be5290f8f973d98"} Oct 31 00:45:05.392331 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:05.392263 5978 scope.go:122] "RemoveContainer" containerID="3e7f5fc1a4aa420bf7835fd8e3db3cdb15648819f65a7c803608d78961f09472" Oct 31 00:45:05.392518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:05.392501 5978 scope.go:122] "RemoveContainer" containerID="cd034b6e6c91b00ee885aa43469f5df1ba638840b7ace9089be5290f8f973d98" Oct 31 00:45:05.392886 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:05.392865 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd_odh-ai-gateway-infra(dd009a89-07a6-4b32-bf9e-9969715eb647)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" Oct 31 00:45:05.394345 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:05.394325 5978 scope.go:122] "RemoveContainer" containerID="3f3ecd7394815714bf788854ce9db958869a70f74523d832926720020e466b66" Oct 31 00:45:05.394708 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:05.394687 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv_odh-ai-gateway-infra(1b946e9a-4b0f-49c7-ac5b-cb39368f56eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" Oct 31 00:45:05.783871 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:05.783774 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:06.405646 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:06.405616 5978 scope.go:122] "RemoveContainer" containerID="cd034b6e6c91b00ee885aa43469f5df1ba638840b7ace9089be5290f8f973d98" Oct 31 00:45:06.406077 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:06.406023 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd_odh-ai-gateway-infra(dd009a89-07a6-4b32-bf9e-9969715eb647)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" Oct 31 00:45:07.536799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:07.536762 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:07.725260 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:07.725223 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:07.975997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:07.975965 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:08.357406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:08.357376 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:09.384433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:09.384393 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:11.064281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:11.064247 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:11.209101 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:11.209064 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:11.538851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:11.538815 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:13.249078 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:13.249038 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:14.418008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:14.417971 5978 scope.go:122] "RemoveContainer" containerID="119f5ca7c535afd00771b1a44bc7313f60fd03fd5a42496e8c7fd45445453534" Oct 31 00:45:15.032990 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:15.032952 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x"] Oct 31 00:45:15.042036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:15.041999 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d28da1e6-7f59x"] Oct 31 00:45:15.075536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:15.075433 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:15.418269 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:15.418236 5978 scope.go:122] "RemoveContainer" containerID="ce814c8ba5548d4c54da36770bb399d7f2177fe1f9d2c29c3d7132a4fa67cee6" Oct 31 00:45:15.516773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:15.516741 5978 generic.go:356] "Generic (PLEG): container finished" podID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerID="5c426adb6ff52cccc644ef7935b52a438b4c13490c61a8a1ce02fd83e2ddc520" exitCode=60 Oct 31 00:45:15.516977 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:15.516809 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" event={"ID":"c8fcba1e-3411-466d-ab12-965d3d34ad44","Type":"ContainerDied","Data":"5c426adb6ff52cccc644ef7935b52a438b4c13490c61a8a1ce02fd83e2ddc520"} Oct 31 00:45:15.516977 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:15.516871 5978 scope.go:122] "RemoveContainer" containerID="119f5ca7c535afd00771b1a44bc7313f60fd03fd5a42496e8c7fd45445453534" Oct 31 00:45:15.517578 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:15.517547 5978 scope.go:122] "RemoveContainer" containerID="5c426adb6ff52cccc644ef7935b52a438b4c13490c61a8a1ce02fd83e2ddc520" Oct 31 00:45:15.518110 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:15.518073 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk_odh-ai-gateway-infra(c8fcba1e-3411-466d-ab12-965d3d34ad44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" Oct 31 00:45:16.437482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:16.437350 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5393f4f-6f8f-488c-93ea-f209802265cd" path="/var/lib/kubelet/pods/d5393f4f-6f8f-488c-93ea-f209802265cd/volumes" Oct 31 00:45:16.542057 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:16.542016 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" event={"ID":"916c10a6-5fe3-4d5f-872e-ab58abaca3eb","Type":"ContainerStarted","Data":"31aa06fa6de2fae8f1779e2e08c057e649a8189f0c4cbfe6bd60d9a1fc11fe67"} Oct 31 00:45:16.585440 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:16.585401 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk"] Oct 31 00:45:16.916613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:16.916584 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:17.027523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.022037 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fcba1e-3411-466d-ab12-965d3d34ad44-kube-api-access-9nqct\" (UniqueName: \"kubernetes.io/projected/c8fcba1e-3411-466d-ab12-965d3d34ad44-kube-api-access-9nqct\") pod \"c8fcba1e-3411-466d-ab12-965d3d34ad44\" (UID: \"c8fcba1e-3411-466d-ab12-965d3d34ad44\") " Oct 31 00:45:17.027523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.022238 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fcba1e-3411-466d-ab12-965d3d34ad44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcba1e-3411-466d-ab12-965d3d34ad44-maas-api-service-ca\") pod \"c8fcba1e-3411-466d-ab12-965d3d34ad44\" (UID: \"c8fcba1e-3411-466d-ab12-965d3d34ad44\") " Oct 31 00:45:17.050476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.032675 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fcba1e-3411-466d-ab12-965d3d34ad44-maas-api-service-ca" pod "c8fcba1e-3411-466d-ab12-965d3d34ad44" (UID: "c8fcba1e-3411-466d-ab12-965d3d34ad44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:17.138617 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.132079 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcba1e-3411-466d-ab12-965d3d34ad44-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:17.291536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.290751 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fcba1e-3411-466d-ab12-965d3d34ad44-kube-api-access-9nqct" pod "c8fcba1e-3411-466d-ab12-965d3d34ad44" (UID: "c8fcba1e-3411-466d-ab12-965d3d34ad44"). InnerVolumeSpecName "kube-api-access-9nqct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:17.339522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.338480 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nqct\" (UniqueName: \"kubernetes.io/projected/c8fcba1e-3411-466d-ab12-965d3d34ad44-kube-api-access-9nqct\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:17.420344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.420306 5978 scope.go:122] "RemoveContainer" containerID="3f3ecd7394815714bf788854ce9db958869a70f74523d832926720020e466b66" Oct 31 00:45:17.422263 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.422234 5978 scope.go:122] "RemoveContainer" containerID="a0371b6d73becdb99258a11c405685b81527783bc756053c9f645d137d10e4a5" Oct 31 00:45:17.471095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.471055 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:17.693480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.693136 5978 generic.go:356] "Generic (PLEG): container finished" podID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerID="31aa06fa6de2fae8f1779e2e08c057e649a8189f0c4cbfe6bd60d9a1fc11fe67" exitCode=60 Oct 31 00:45:17.693480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.693294 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" event={"ID":"916c10a6-5fe3-4d5f-872e-ab58abaca3eb","Type":"ContainerDied","Data":"31aa06fa6de2fae8f1779e2e08c057e649a8189f0c4cbfe6bd60d9a1fc11fe67"} Oct 31 00:45:17.693480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.693353 5978 scope.go:122] "RemoveContainer" containerID="ce814c8ba5548d4c54da36770bb399d7f2177fe1f9d2c29c3d7132a4fa67cee6" Oct 31 00:45:17.717477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.693920 5978 scope.go:122] "RemoveContainer" containerID="31aa06fa6de2fae8f1779e2e08c057e649a8189f0c4cbfe6bd60d9a1fc11fe67" Oct 31 00:45:17.717477 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:17.694392 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4_odh-ai-gateway-infra(916c10a6-5fe3-4d5f-872e-ab58abaca3eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" Oct 31 00:45:17.770491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.770053 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4"] Oct 31 00:45:17.808507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.794270 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" event={"ID":"c8fcba1e-3411-466d-ab12-965d3d34ad44","Type":"ContainerDied","Data":"fb5cdc9c70f13cca9457c505c72d2eec7961998a3366a8211eff4b52051445d0"} Oct 31 00:45:17.808507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.794483 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk" Oct 31 00:45:17.956539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:17.947888 5978 scope.go:122] "RemoveContainer" containerID="5c426adb6ff52cccc644ef7935b52a438b4c13490c61a8a1ce02fd83e2ddc520" Oct 31 00:45:18.278779 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:18.278742 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk"] Oct 31 00:45:18.559288 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:18.537887 5978 scope.go:122] "RemoveContainer" containerID="cd034b6e6c91b00ee885aa43469f5df1ba638840b7ace9089be5290f8f973d98" Oct 31 00:45:18.729478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:18.669873 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.762172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:18.762124 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.879951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:18.876548 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.905479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:18.896171 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.938633 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:18.938582 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:18.964249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:18.964176 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.064500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.064438 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a9461dc0-29890125-xbmpk"] Oct 31 00:45:19.170797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.170698 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:45:19.463289 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.463251 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:45:19.539480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.515180 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" event={"ID":"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9","Type":"ContainerStarted","Data":"51ac78300103a6d9ad75bc35ec3d06198b98054e74d265afd23790571fcf0a0c"} Oct 31 00:45:19.558552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.541468 5978 scope.go:122] "RemoveContainer" containerID="51ac78300103a6d9ad75bc35ec3d06198b98054e74d265afd23790571fcf0a0c" Oct 31 00:45:19.601479 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:19.569149 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-mw9vj_odh-ai-gateway-infra(4a243edd-e5c8-4bdf-8174-9a9f5542f5d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" Oct 31 00:45:19.711927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.711829 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" event={"ID":"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb","Type":"ContainerStarted","Data":"4e6cd6809a829bcdd9c4ee4dd00e440f145ccbc657afe020ae2036d4a33a027f"} Oct 31 00:45:19.750665 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.712738 5978 scope.go:122] "RemoveContainer" containerID="4e6cd6809a829bcdd9c4ee4dd00e440f145ccbc657afe020ae2036d4a33a027f" Oct 31 00:45:19.925479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.715334 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.925479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.800529 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:45:19.925479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.922387 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:45:19.925479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.922534 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:45:19.925479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.896573 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.925479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.925031 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:45:19.956479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.944576 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:19.994547 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:19.957172 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:45:19.994547 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:19.967066 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv_odh-ai-gateway-infra(1b946e9a-4b0f-49c7-ac5b-cb39368f56eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" Oct 31 00:45:20.177108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:20.174636 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:20.316560 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:20.275374 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:45:20.316560 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:20.290116 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:20.563530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:20.558027 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:45:20.927484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:20.911813 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:20.961130 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:20.947794 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:20.982172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:20.979037 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:20.982172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:20.980641 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:45:20.982586 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:20.982534 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:45:21.035842 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.024682 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:21.228082 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.196571 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:45:21.230437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.230339 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" path="/var/lib/kubelet/pods/c8fcba1e-3411-466d-ab12-965d3d34ad44/volumes" Oct 31 00:45:21.231584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.231549 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:45:21.231793 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.231781 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" Oct 31 00:45:21.231903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.231893 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj"] Oct 31 00:45:21.232042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.232033 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 00:45:21.270438 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.270392 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 00:45:21.392869 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.390038 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 00:45:21.450762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.450729 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:45:21.458482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.458431 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv"] Oct 31 00:45:21.458958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.458869 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 00:45:21.460700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.460672 5978 generic.go:356] "Generic (PLEG): container finished" podID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerID="51ac78300103a6d9ad75bc35ec3d06198b98054e74d265afd23790571fcf0a0c" exitCode=60 Oct 31 00:45:21.461036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.461017 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" event={"ID":"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9","Type":"ContainerDied","Data":"51ac78300103a6d9ad75bc35ec3d06198b98054e74d265afd23790571fcf0a0c"} Oct 31 00:45:21.461176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.461164 5978 scope.go:122] "RemoveContainer" containerID="a0371b6d73becdb99258a11c405685b81527783bc756053c9f645d137d10e4a5" Oct 31 00:45:21.583721 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.582565 5978 generic.go:356] "Generic (PLEG): container finished" podID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerID="4e6cd6809a829bcdd9c4ee4dd00e440f145ccbc657afe020ae2036d4a33a027f" exitCode=60 Oct 31 00:45:21.583721 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.583029 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" event={"ID":"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb","Type":"ContainerDied","Data":"4e6cd6809a829bcdd9c4ee4dd00e440f145ccbc657afe020ae2036d4a33a027f"} Oct 31 00:45:21.589270 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.587855 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:21.590884 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.590852 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-maas-api-service-ca\") pod \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\" (UID: \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\") " Oct 31 00:45:21.592703 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.592675 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-maas-api-service-ca" pod "916c10a6-5fe3-4d5f-872e-ab58abaca3eb" (UID: "916c10a6-5fe3-4d5f-872e-ab58abaca3eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:21.776342 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.775883 5978 scope.go:122] "RemoveContainer" containerID="3f3ecd7394815714bf788854ce9db958869a70f74523d832926720020e466b66" Oct 31 00:45:21.781207 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.779098 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-kube-api-access-gbgfw\" (UniqueName: \"kubernetes.io/projected/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-kube-api-access-gbgfw\") pod \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\" (UID: \"916c10a6-5fe3-4d5f-872e-ab58abaca3eb\") " Oct 31 00:45:21.805199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.790752 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:21.847478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.847411 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-kube-api-access-gbgfw" pod "916c10a6-5fe3-4d5f-872e-ab58abaca3eb" (UID: "916c10a6-5fe3-4d5f-872e-ab58abaca3eb"). InnerVolumeSpecName "kube-api-access-gbgfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:21.911479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:21.894306 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbgfw\" (UniqueName: \"kubernetes.io/projected/916c10a6-5fe3-4d5f-872e-ab58abaca3eb-kube-api-access-gbgfw\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:22.101953 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:22.101839 5978 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd009a89_07a6_4b32_bf9e_9969715eb647.slice/crio-conmon-c52623634f40eaedcd966e0fa65d6edb03c2ad14a609cdd8db499956cc820cd0.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:45:22.197270 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.197238 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:22.295240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.295207 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:22.351149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.351103 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-maas-api-service-ca\") pod \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\" (UID: \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\") " Oct 31 00:45:22.351548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.351524 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-kube-api-access-trvgv\" (UniqueName: \"kubernetes.io/projected/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-kube-api-access-trvgv\") pod \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\" (UID: \"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9\") " Oct 31 00:45:22.353257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.353184 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-maas-api-service-ca" pod "4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" (UID: "4a243edd-e5c8-4bdf-8174-9a9f5542f5d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:22.355417 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.355386 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-kube-api-access-trvgv" pod "4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" (UID: "4a243edd-e5c8-4bdf-8174-9a9f5542f5d9"). InnerVolumeSpecName "kube-api-access-trvgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:22.454009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.453958 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-maas-api-service-ca\") pod \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\" (UID: \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\") " Oct 31 00:45:22.454308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.454285 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-maas-api-service-ca" pod "1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" (UID: "1b946e9a-4b0f-49c7-ac5b-cb39368f56eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:22.454360 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.454322 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-kube-api-access-wmlhk\" (UniqueName: \"kubernetes.io/projected/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-kube-api-access-wmlhk\") pod \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\" (UID: \"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb\") " Oct 31 00:45:22.455556 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.455530 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:22.455656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.455562 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trvgv\" (UniqueName: \"kubernetes.io/projected/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-kube-api-access-trvgv\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:22.455656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.455577 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:22.456526 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.456497 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-kube-api-access-wmlhk" pod "1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" (UID: "1b946e9a-4b0f-49c7-ac5b-cb39368f56eb"). InnerVolumeSpecName "kube-api-access-wmlhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:22.524078 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.524039 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:22.557784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.557751 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmlhk\" (UniqueName: \"kubernetes.io/projected/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb-kube-api-access-wmlhk\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:22.594033 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.594003 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" Oct 31 00:45:22.594235 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.594006 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4" event={"ID":"916c10a6-5fe3-4d5f-872e-ab58abaca3eb","Type":"ContainerDied","Data":"eb2eed21d223e417d91af778bbaf1ec656a1777fde6a25034ac0207b6cebed62"} Oct 31 00:45:22.594235 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.594132 5978 scope.go:122] "RemoveContainer" containerID="31aa06fa6de2fae8f1779e2e08c057e649a8189f0c4cbfe6bd60d9a1fc11fe67" Oct 31 00:45:22.595679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.595661 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" Oct 31 00:45:22.595822 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.595694 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj" event={"ID":"4a243edd-e5c8-4bdf-8174-9a9f5542f5d9","Type":"ContainerDied","Data":"55f4d51300aa7cbda9b3c84588a5343f246b21213d4c940aa62b7d432145e173"} Oct 31 00:45:22.598362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.598338 5978 generic.go:356] "Generic (PLEG): container finished" podID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerID="c52623634f40eaedcd966e0fa65d6edb03c2ad14a609cdd8db499956cc820cd0" exitCode=60 Oct 31 00:45:22.598508 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.598398 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" event={"ID":"dd009a89-07a6-4b32-bf9e-9969715eb647","Type":"ContainerDied","Data":"c52623634f40eaedcd966e0fa65d6edb03c2ad14a609cdd8db499956cc820cd0"} Oct 31 00:45:22.598992 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.598969 5978 scope.go:122] "RemoveContainer" containerID="c52623634f40eaedcd966e0fa65d6edb03c2ad14a609cdd8db499956cc820cd0" Oct 31 00:45:22.599407 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:22.599386 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd_odh-ai-gateway-infra(dd009a89-07a6-4b32-bf9e-9969715eb647)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" Oct 31 00:45:22.600108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.600082 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" event={"ID":"1b946e9a-4b0f-49c7-ac5b-cb39368f56eb","Type":"ContainerDied","Data":"016cf78196d5ad4edfbc9902b4ee0ce4f3f9a4695a49065609f6645cbb0612fe"} Oct 31 00:45:22.600208 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.600175 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv" Oct 31 00:45:22.609615 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.609591 5978 scope.go:122] "RemoveContainer" containerID="51ac78300103a6d9ad75bc35ec3d06198b98054e74d265afd23790571fcf0a0c" Oct 31 00:45:22.621490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.621465 5978 scope.go:122] "RemoveContainer" containerID="cd034b6e6c91b00ee885aa43469f5df1ba638840b7ace9089be5290f8f973d98" Oct 31 00:45:22.636832 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.636806 5978 scope.go:122] "RemoveContainer" containerID="4e6cd6809a829bcdd9c4ee4dd00e440f145ccbc657afe020ae2036d4a33a027f" Oct 31 00:45:22.654423 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.654387 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj"] Oct 31 00:45:22.660152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.660115 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-mw9vj"] Oct 31 00:45:22.693234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.693193 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4"] Oct 31 00:45:22.698881 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.698848 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0d8bae-29890125-gvfp4"] Oct 31 00:45:22.761530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.761494 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv"] Oct 31 00:45:22.767338 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:22.767299 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890125-ghnmv"] Oct 31 00:45:23.747449 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:23.747412 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd"] Oct 31 00:45:23.925585 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:23.925550 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:23.965373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:23.965336 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:23.985966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:23.985931 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd009a89-07a6-4b32-bf9e-9969715eb647-kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/dd009a89-07a6-4b32-bf9e-9969715eb647-kube-api-access-9nqkz\") pod \"dd009a89-07a6-4b32-bf9e-9969715eb647\" (UID: \"dd009a89-07a6-4b32-bf9e-9969715eb647\") " Oct 31 00:45:23.986267 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:23.986246 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd009a89-07a6-4b32-bf9e-9969715eb647-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd009a89-07a6-4b32-bf9e-9969715eb647-maas-api-service-ca\") pod \"dd009a89-07a6-4b32-bf9e-9969715eb647\" (UID: \"dd009a89-07a6-4b32-bf9e-9969715eb647\") " Oct 31 00:45:23.986640 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:23.986617 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd009a89-07a6-4b32-bf9e-9969715eb647-maas-api-service-ca" pod "dd009a89-07a6-4b32-bf9e-9969715eb647" (UID: "dd009a89-07a6-4b32-bf9e-9969715eb647"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:23.986957 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:23.986941 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd009a89-07a6-4b32-bf9e-9969715eb647-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:23.988672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:23.988639 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd009a89-07a6-4b32-bf9e-9969715eb647-kube-api-access-9nqkz" pod "dd009a89-07a6-4b32-bf9e-9969715eb647" (UID: "dd009a89-07a6-4b32-bf9e-9969715eb647"). InnerVolumeSpecName "kube-api-access-9nqkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:24.088885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.088847 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/dd009a89-07a6-4b32-bf9e-9969715eb647-kube-api-access-9nqkz\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:24.432089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.432047 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" path="/var/lib/kubelet/pods/1b946e9a-4b0f-49c7-ac5b-cb39368f56eb/volumes" Oct 31 00:45:24.432618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.432602 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" path="/var/lib/kubelet/pods/4a243edd-e5c8-4bdf-8174-9a9f5542f5d9/volumes" Oct 31 00:45:24.433048 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.433034 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" path="/var/lib/kubelet/pods/916c10a6-5fe3-4d5f-872e-ab58abaca3eb/volumes" Oct 31 00:45:24.629611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.629578 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" Oct 31 00:45:24.629611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.629586 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd" event={"ID":"dd009a89-07a6-4b32-bf9e-9969715eb647","Type":"ContainerDied","Data":"4f3069c4652a20d3e58dc5991f319022a47f46369828abad29f54a5e968bf97e"} Oct 31 00:45:24.629882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.629656 5978 scope.go:122] "RemoveContainer" containerID="c52623634f40eaedcd966e0fa65d6edb03c2ad14a609cdd8db499956cc820cd0" Oct 31 00:45:24.686624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.686527 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd"] Oct 31 00:45:24.692522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.692467 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890125-n4nmd"] Oct 31 00:45:24.725408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.725368 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt"] Oct 31 00:45:24.726286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726263 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerName="cleanup" Oct 31 00:45:24.726286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726286 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726309 5978 cpu_manager.go:414] "containerMap: removing container" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726316 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726325 5978 cpu_manager.go:414] "containerMap: removing container" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726333 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726349 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726356 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726376 5978 cpu_manager.go:414] "containerMap: removing container" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726382 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726393 5978 cpu_manager.go:414] "containerMap: removing container" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726400 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726412 5978 cpu_manager.go:414] "containerMap: removing container" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726419 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726440 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726448 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726490 5978 cpu_manager.go:414] "containerMap: removing container" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerName="cleanup" Oct 31 00:45:24.726506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726505 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726523 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726530 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726773 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726789 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726799 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726814 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726825 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726834 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726844 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726853 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726861 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726874 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerName="cleanup" Oct 31 00:45:24.727253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.726883 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerName="cleanup" Oct 31 00:45:24.732716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.732688 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:24.750056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.749971 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:45:24.750056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.750034 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:24.752724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.752697 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt"] Oct 31 00:45:24.821297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.821258 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7qlc\" (UniqueName: \"kubernetes.io/projected/ca827deb-869b-4423-a9ee-857c219e360d-kube-api-access-d7qlc\") pod \"maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:24.821568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.821405 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:24.821568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.821495 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:24.924379 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.924337 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qlc\" (UniqueName: \"kubernetes.io/projected/ca827deb-869b-4423-a9ee-857c219e360d-kube-api-access-d7qlc\") pod \"maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:24.924865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.924825 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:24.925104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.924879 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:24.926821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.926770 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:24.928162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.928136 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:24.947347 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:24.947272 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qlc\" (UniqueName: \"kubernetes.io/projected/ca827deb-869b-4423-a9ee-857c219e360d-kube-api-access-d7qlc\") pod \"maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:25.054792 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:25.054755 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:25.371821 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:45:25.371787 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca827deb_869b_4423_a9ee_857c219e360d.slice/crio-8644ea67b5871297c3345e44e7272878591336303005a19ea4adf280a15cd95a WatchSource:0}: Error finding container 8644ea67b5871297c3345e44e7272878591336303005a19ea4adf280a15cd95a: Status 404 returned error can't find the container with id 8644ea67b5871297c3345e44e7272878591336303005a19ea4adf280a15cd95a Oct 31 00:45:25.373172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:25.373146 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt"] Oct 31 00:45:25.641161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:25.641125 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" event={"ID":"ca827deb-869b-4423-a9ee-857c219e360d","Type":"ContainerStarted","Data":"8644ea67b5871297c3345e44e7272878591336303005a19ea4adf280a15cd95a"} Oct 31 00:45:26.038276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:26.038166 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw"] Oct 31 00:45:26.045425 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:26.045389 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc631a8-txxlw"] Oct 31 00:45:26.427972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:26.427934 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3ea689-f4a7-4b83-afcc-7446124457ae" path="/var/lib/kubelet/pods/5f3ea689-f4a7-4b83-afcc-7446124457ae/volumes" Oct 31 00:45:26.428589 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:26.428573 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" path="/var/lib/kubelet/pods/dd009a89-07a6-4b32-bf9e-9969715eb647/volumes" Oct 31 00:45:26.656986 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:26.656877 5978 generic.go:356] "Generic (PLEG): container finished" podID="ca827deb-869b-4423-a9ee-857c219e360d" containerID="02c4267aed3b09e1221019502f5f106d26ca750a29720718b64313872ba13e03" exitCode=0 Oct 31 00:45:26.657179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:26.657117 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" event={"ID":"ca827deb-869b-4423-a9ee-857c219e360d","Type":"ContainerDied","Data":"02c4267aed3b09e1221019502f5f106d26ca750a29720718b64313872ba13e03"} Oct 31 00:45:26.844219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:26.844182 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:27.844251 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:27.844217 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:27.978989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:27.978888 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-openshift-ca\") pod \"ca827deb-869b-4423-a9ee-857c219e360d\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " Oct 31 00:45:27.979154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:27.979078 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca827deb-869b-4423-a9ee-857c219e360d-kube-api-access-d7qlc\" (UniqueName: \"kubernetes.io/projected/ca827deb-869b-4423-a9ee-857c219e360d-kube-api-access-d7qlc\") pod \"ca827deb-869b-4423-a9ee-857c219e360d\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " Oct 31 00:45:27.979200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:27.979168 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-xks-ca\") pod \"ca827deb-869b-4423-a9ee-857c219e360d\" (UID: \"ca827deb-869b-4423-a9ee-857c219e360d\") " Oct 31 00:45:27.979286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:27.979260 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-openshift-ca" pod "ca827deb-869b-4423-a9ee-857c219e360d" (UID: "ca827deb-869b-4423-a9ee-857c219e360d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:27.979800 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:27.979775 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:27.981576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:27.981535 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-xks-ca" pod "ca827deb-869b-4423-a9ee-857c219e360d" (UID: "ca827deb-869b-4423-a9ee-857c219e360d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:27.981576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:27.981547 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca827deb-869b-4423-a9ee-857c219e360d-kube-api-access-d7qlc" pod "ca827deb-869b-4423-a9ee-857c219e360d" (UID: "ca827deb-869b-4423-a9ee-857c219e360d"). InnerVolumeSpecName "kube-api-access-d7qlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:28.081138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.081098 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7qlc\" (UniqueName: \"kubernetes.io/projected/ca827deb-869b-4423-a9ee-857c219e360d-kube-api-access-d7qlc\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:28.081138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.081134 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca827deb-869b-4423-a9ee-857c219e360d-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:28.532473 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.532417 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb"] Oct 31 00:45:28.533305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533285 5978 cpu_manager.go:414] "containerMap: removing container" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerName="cleanup" Oct 31 00:45:28.533305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533304 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerName="cleanup" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533318 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerName="cleanup" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533323 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerName="cleanup" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533340 5978 cpu_manager.go:414] "containerMap: removing container" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerName="cleanup" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533345 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerName="cleanup" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533355 5978 cpu_manager.go:414] "containerMap: removing container" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerName="cleanup" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533359 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd009a89-07a6-4b32-bf9e-9969715eb647" containerName="cleanup" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533382 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerName="cleanup" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533387 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerName="cleanup" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533405 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ca827deb-869b-4423-a9ee-857c219e360d" containerName="revoke-keys" Oct 31 00:45:28.533442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533409 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca827deb-869b-4423-a9ee-857c219e360d" containerName="revoke-keys" Oct 31 00:45:28.533786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533551 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fcba1e-3411-466d-ab12-965d3d34ad44" containerName="cleanup" Oct 31 00:45:28.533786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533562 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a243edd-e5c8-4bdf-8174-9a9f5542f5d9" containerName="cleanup" Oct 31 00:45:28.533786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533570 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b946e9a-4b0f-49c7-ac5b-cb39368f56eb" containerName="cleanup" Oct 31 00:45:28.533786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533577 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="916c10a6-5fe3-4d5f-872e-ab58abaca3eb" containerName="cleanup" Oct 31 00:45:28.533786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.533593 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca827deb-869b-4423-a9ee-857c219e360d" containerName="revoke-keys" Oct 31 00:45:28.537751 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.537726 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.559698 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.559661 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb"] Oct 31 00:45:28.589206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.589172 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96814a5d-6vtwb\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.589385 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.589323 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrdl\" (UniqueName: \"kubernetes.io/projected/072b4b37-7db9-4dce-a3b7-41eb4a00022d-kube-api-access-zdrdl\") pod \"maas-api-revoke-sub-96814a5d-6vtwb\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.589555 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.589538 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96814a5d-6vtwb\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.678552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.678522 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" Oct 31 00:45:28.678724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.678519 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt" event={"ID":"ca827deb-869b-4423-a9ee-857c219e360d","Type":"ContainerDied","Data":"8644ea67b5871297c3345e44e7272878591336303005a19ea4adf280a15cd95a"} Oct 31 00:45:28.678724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.678663 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8644ea67b5871297c3345e44e7272878591336303005a19ea4adf280a15cd95a" Oct 31 00:45:28.691079 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.690991 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96814a5d-6vtwb\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.691242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.691147 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96814a5d-6vtwb\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.691242 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.691236 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrdl\" (UniqueName: \"kubernetes.io/projected/072b4b37-7db9-4dce-a3b7-41eb4a00022d-kube-api-access-zdrdl\") pod \"maas-api-revoke-sub-96814a5d-6vtwb\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.691818 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.691793 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96814a5d-6vtwb\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.693686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.693666 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96814a5d-6vtwb\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.713523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.713494 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrdl\" (UniqueName: \"kubernetes.io/projected/072b4b37-7db9-4dce-a3b7-41eb4a00022d-kube-api-access-zdrdl\") pod \"maas-api-revoke-sub-96814a5d-6vtwb\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.856151 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.856105 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:28.888754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:28.886174 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:29.230626 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:45:29.230546 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072b4b37_7db9_4dce_a3b7_41eb4a00022d.slice/crio-ce06b2c521ccbf6d34eff62423c9e16f4ae5483e6d81bc9622e9cadc9cba0e00 WatchSource:0}: Error finding container ce06b2c521ccbf6d34eff62423c9e16f4ae5483e6d81bc9622e9cadc9cba0e00: Status 404 returned error can't find the container with id ce06b2c521ccbf6d34eff62423c9e16f4ae5483e6d81bc9622e9cadc9cba0e00 Oct 31 00:45:29.232464 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:29.232423 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb"] Oct 31 00:45:29.690737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:29.690690 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" event={"ID":"072b4b37-7db9-4dce-a3b7-41eb4a00022d","Type":"ContainerStarted","Data":"3b038947aa52e5f8b4cf445ed2e498d6c567deec5d27a828ebf57d9f0e586ca6"} Oct 31 00:45:29.690922 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:29.690757 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" event={"ID":"072b4b37-7db9-4dce-a3b7-41eb4a00022d","Type":"ContainerStarted","Data":"ce06b2c521ccbf6d34eff62423c9e16f4ae5483e6d81bc9622e9cadc9cba0e00"} Oct 31 00:45:29.730929 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:29.730871 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" podStartSLOduration=1.730856436 podStartE2EDuration="1.730856436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:29.727167228 +0000 UTC m=+1547.425280994" watchObservedRunningTime="2026-10-31 00:45:29.730856436 +0000 UTC m=+1547.428970215" Oct 31 00:45:30.598356 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:30.598309 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:30.704962 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:30.704927 5978 generic.go:356] "Generic (PLEG): container finished" podID="072b4b37-7db9-4dce-a3b7-41eb4a00022d" containerID="3b038947aa52e5f8b4cf445ed2e498d6c567deec5d27a828ebf57d9f0e586ca6" exitCode=0 Oct 31 00:45:30.705158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:30.705008 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" event={"ID":"072b4b37-7db9-4dce-a3b7-41eb4a00022d","Type":"ContainerDied","Data":"3b038947aa52e5f8b4cf445ed2e498d6c567deec5d27a828ebf57d9f0e586ca6"} Oct 31 00:45:31.037139 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.037041 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw"] Oct 31 00:45:31.043730 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.043693 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe0f1be-h5bgw"] Oct 31 00:45:31.910794 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.910760 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:31.989144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.989037 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-openshift-ca\") pod \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " Oct 31 00:45:31.989144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.989090 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-xks-ca\") pod \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " Oct 31 00:45:31.989144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.989122 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072b4b37-7db9-4dce-a3b7-41eb4a00022d-kube-api-access-zdrdl\" (UniqueName: \"kubernetes.io/projected/072b4b37-7db9-4dce-a3b7-41eb4a00022d-kube-api-access-zdrdl\") pod \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\" (UID: \"072b4b37-7db9-4dce-a3b7-41eb4a00022d\") " Oct 31 00:45:31.989585 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.989554 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-openshift-ca" pod "072b4b37-7db9-4dce-a3b7-41eb4a00022d" (UID: "072b4b37-7db9-4dce-a3b7-41eb4a00022d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:31.989987 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.989956 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:31.992065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.992027 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072b4b37-7db9-4dce-a3b7-41eb4a00022d-kube-api-access-zdrdl" pod "072b4b37-7db9-4dce-a3b7-41eb4a00022d" (UID: "072b4b37-7db9-4dce-a3b7-41eb4a00022d"). InnerVolumeSpecName "kube-api-access-zdrdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:31.992173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:31.992091 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-xks-ca" pod "072b4b37-7db9-4dce-a3b7-41eb4a00022d" (UID: "072b4b37-7db9-4dce-a3b7-41eb4a00022d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:32.092065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:32.092019 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/072b4b37-7db9-4dce-a3b7-41eb4a00022d-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:32.092065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:32.092058 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdrdl\" (UniqueName: \"kubernetes.io/projected/072b4b37-7db9-4dce-a3b7-41eb4a00022d-kube-api-access-zdrdl\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:32.322973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:32.322906 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:32.436517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:32.434794 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8bb4b0-f452-43a3-9c1e-26419dbc059f" path="/var/lib/kubelet/pods/7e8bb4b0-f452-43a3-9c1e-26419dbc059f/volumes" Oct 31 00:45:32.724257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:32.724165 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" Oct 31 00:45:32.724257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:32.724193 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb" event={"ID":"072b4b37-7db9-4dce-a3b7-41eb4a00022d","Type":"ContainerDied","Data":"ce06b2c521ccbf6d34eff62423c9e16f4ae5483e6d81bc9622e9cadc9cba0e00"} Oct 31 00:45:32.724524 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:32.724258 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce06b2c521ccbf6d34eff62423c9e16f4ae5483e6d81bc9622e9cadc9cba0e00" Oct 31 00:45:33.184657 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:33.184614 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:45:33.750851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.750817 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk"] Oct 31 00:45:33.751720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.751696 5978 cpu_manager.go:414] "containerMap: removing container" podUID="072b4b37-7db9-4dce-a3b7-41eb4a00022d" containerName="revoke-keys" Oct 31 00:45:33.751720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.751720 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072b4b37-7db9-4dce-a3b7-41eb4a00022d" containerName="revoke-keys" Oct 31 00:45:33.751972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.751957 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="072b4b37-7db9-4dce-a3b7-41eb4a00022d" containerName="revoke-keys" Oct 31 00:45:33.756042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.756017 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.775745 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.775713 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk"] Oct 31 00:45:33.779407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.779376 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:33.779599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.779489 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:45:33.821432 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.821390 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whprs\" (UniqueName: \"kubernetes.io/projected/8da44a50-8b2e-4669-a550-f278371c4c6d-kube-api-access-whprs\") pod \"maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.821672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.821649 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.821855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.821834 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.923558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.923513 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.923753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.923695 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.923951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.923931 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whprs\" (UniqueName: \"kubernetes.io/projected/8da44a50-8b2e-4669-a550-f278371c4c6d-kube-api-access-whprs\") pod \"maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.924407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.924379 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.926184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.926160 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.945774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.945734 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whprs\" (UniqueName: \"kubernetes.io/projected/8da44a50-8b2e-4669-a550-f278371c4c6d-kube-api-access-whprs\") pod \"maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:33.947110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:33.947026 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:34.075946 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:34.072060 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:34.468671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:34.468625 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk"] Oct 31 00:45:34.747624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:34.747584 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" event={"ID":"8da44a50-8b2e-4669-a550-f278371c4c6d","Type":"ContainerStarted","Data":"4a74c9525ba9f95660a48b52ae4cc07534bece15f43bdcb1fc5c1237d5fc9805"} Oct 31 00:45:34.845694 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:34.845657 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:35.058251 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:35.058165 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:35.203549 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:35.203513 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:35.776747 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:35.776641 5978 generic.go:356] "Generic (PLEG): container finished" podID="8da44a50-8b2e-4669-a550-f278371c4c6d" containerID="b29a7126773042fc537ba17401eb881fa48adefd2c1cd7e659611386ebd5a002" exitCode=0 Oct 31 00:45:35.776747 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:35.776728 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" event={"ID":"8da44a50-8b2e-4669-a550-f278371c4c6d","Type":"ContainerDied","Data":"b29a7126773042fc537ba17401eb881fa48adefd2c1cd7e659611386ebd5a002"} Oct 31 00:45:35.780393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:35.780359 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:35.939085 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:35.938800 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:36.744437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:36.744395 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.085210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.085170 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:37.159146 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.159107 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da44a50-8b2e-4669-a550-f278371c4c6d-kube-api-access-whprs\" (UniqueName: \"kubernetes.io/projected/8da44a50-8b2e-4669-a550-f278371c4c6d-kube-api-access-whprs\") pod \"8da44a50-8b2e-4669-a550-f278371c4c6d\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " Oct 31 00:45:37.159398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.159372 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-openshift-ca\") pod \"8da44a50-8b2e-4669-a550-f278371c4c6d\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " Oct 31 00:45:37.159551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.159493 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-xks-ca\") pod \"8da44a50-8b2e-4669-a550-f278371c4c6d\" (UID: \"8da44a50-8b2e-4669-a550-f278371c4c6d\") " Oct 31 00:45:37.161361 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.161328 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-openshift-ca" pod "8da44a50-8b2e-4669-a550-f278371c4c6d" (UID: "8da44a50-8b2e-4669-a550-f278371c4c6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:37.163243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.163212 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-xks-ca" pod "8da44a50-8b2e-4669-a550-f278371c4c6d" (UID: "8da44a50-8b2e-4669-a550-f278371c4c6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:37.172731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.172682 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da44a50-8b2e-4669-a550-f278371c4c6d-kube-api-access-whprs" pod "8da44a50-8b2e-4669-a550-f278371c4c6d" (UID: "8da44a50-8b2e-4669-a550-f278371c4c6d"). InnerVolumeSpecName "kube-api-access-whprs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:37.268523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.268487 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whprs\" (UniqueName: \"kubernetes.io/projected/8da44a50-8b2e-4669-a550-f278371c4c6d-kube-api-access-whprs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:37.268523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.268522 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:37.268846 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.268541 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da44a50-8b2e-4669-a550-f278371c4c6d-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:37.394728 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.394689 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.616314 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.616280 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.801421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.801333 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" Oct 31 00:45:37.801421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.801335 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk" event={"ID":"8da44a50-8b2e-4669-a550-f278371c4c6d","Type":"ContainerDied","Data":"4a74c9525ba9f95660a48b52ae4cc07534bece15f43bdcb1fc5c1237d5fc9805"} Oct 31 00:45:37.801667 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.801427 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a74c9525ba9f95660a48b52ae4cc07534bece15f43bdcb1fc5c1237d5fc9805" Oct 31 00:45:37.918885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.918844 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms"] Oct 31 00:45:37.920360 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.920329 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8da44a50-8b2e-4669-a550-f278371c4c6d" containerName="revoke-keys" Oct 31 00:45:37.920360 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.920361 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da44a50-8b2e-4669-a550-f278371c4c6d" containerName="revoke-keys" Oct 31 00:45:37.920716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.920697 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da44a50-8b2e-4669-a550-f278371c4c6d" containerName="revoke-keys" Oct 31 00:45:37.926098 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.926070 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:37.945543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.939323 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c2cd8283-openshift-default-dockercfg-zrtwf\"" Oct 31 00:45:37.949203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:37.948838 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms"] Oct 31 00:45:38.000511 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.000416 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:38.083870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.083818 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194d8e50-d32e-492e-a9ad-66d15823d000-istiod-ca-cert\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.084092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.084037 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-credential-socket\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.084234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.084215 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194d8e50-d32e-492e-a9ad-66d15823d000-istio-podinfo\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.084466 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.084435 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-envoy\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.084595 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.084576 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-certs\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.084656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.084643 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-data\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.084713 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.084673 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnj5\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-kube-api-access-gtnj5\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.084774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.084743 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-istio-token\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.084925 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.084908 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-socket\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.168475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.168374 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:38.186590 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.186542 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-certs\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.186786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.186626 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-data\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.186786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.186653 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnj5\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-kube-api-access-gtnj5\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.186786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.186716 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-istio-token\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.187044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.187023 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-certs\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.187368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.187347 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-data\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.187582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.187554 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-socket\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.187743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.187728 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194d8e50-d32e-492e-a9ad-66d15823d000-istiod-ca-cert\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.187816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.187784 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-credential-socket\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.187913 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.187886 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-socket\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.187972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.187944 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194d8e50-d32e-492e-a9ad-66d15823d000-istio-podinfo\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.188095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.188078 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-envoy\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.188353 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.188336 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194d8e50-d32e-492e-a9ad-66d15823d000-istiod-ca-cert\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.188651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.188582 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-credential-socket\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.190873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.190833 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-envoy\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.192759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.192732 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194d8e50-d32e-492e-a9ad-66d15823d000-istio-podinfo\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.208788 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.208756 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-istio-token\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.209077 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.209052 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnj5\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-kube-api-access-gtnj5\") pod \"e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.256115 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.256082 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:38.616675 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:45:38.616637 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194d8e50_d32e_492e_a9ad_66d15823d000.slice/crio-eb15ef408959f210875d4e9bc21f86b7735f3aa29b7526078e3710bb4376e063 WatchSource:0}: Error finding container eb15ef408959f210875d4e9bc21f86b7735f3aa29b7526078e3710bb4376e063: Status 404 returned error can't find the container with id eb15ef408959f210875d4e9bc21f86b7735f3aa29b7526078e3710bb4376e063 Oct 31 00:45:38.618433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.618404 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms"] Oct 31 00:45:38.619868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.619833 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:38.619969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.619921 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:38.620007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.619966 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:38.816263 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.816213 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" event={"ID":"194d8e50-d32e-492e-a9ad-66d15823d000","Type":"ContainerStarted","Data":"a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85"} Oct 31 00:45:38.816420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.816282 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" event={"ID":"194d8e50-d32e-492e-a9ad-66d15823d000","Type":"ContainerStarted","Data":"eb15ef408959f210875d4e9bc21f86b7735f3aa29b7526078e3710bb4376e063"} Oct 31 00:45:38.862398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:38.862291 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" podStartSLOduration=1.862274328 podStartE2EDuration="1.862274328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:38.858053717 +0000 UTC m=+1556.556167491" watchObservedRunningTime="2026-10-31 00:45:38.862274328 +0000 UTC m=+1556.560388093" Oct 31 00:45:39.054359 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:39.054324 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:39.256680 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:39.256594 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:39.262977 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:39.262947 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:39.826465 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:39.826414 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:39.827415 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:39.827394 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:45:39.938707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:39.938667 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:40.039533 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:40.039447 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl"] Oct 31 00:45:40.045791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:40.045760 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccb04086-lj4zl"] Oct 31 00:45:40.427991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:40.427952 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4385d879-e04f-47cf-ac7c-607dc611c9d2" path="/var/lib/kubelet/pods/4385d879-e04f-47cf-ac7c-607dc611c9d2/volumes" Oct 31 00:45:40.623660 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:40.623619 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:40.825384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:40.825340 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:42.562274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:42.562237 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:43.559980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:43.559942 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:43.732153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:43.732114 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:44.894875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:44.894728 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:45.041806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:45.041708 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:45.261987 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:45.261948 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:45.527869 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:45.527830 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:46.377911 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:46.377874 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:46.520829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:46.520789 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:46.884059 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:46.884012 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:47.261530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:47.259470 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:47.414510 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:47.414437 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt"] Oct 31 00:45:47.426049 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:47.423517 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9461dc0-nw5zt"] Oct 31 00:45:47.722173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:47.722138 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:47.874195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:47.874158 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:48.036398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:48.036005 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w"] Oct 31 00:45:48.053674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:48.053624 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771d076e-8td6w"] Oct 31 00:45:48.428673 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:48.428639 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3d3426-b2ce-49c5-849b-e34598d605e6" path="/var/lib/kubelet/pods/bb3d3426-b2ce-49c5-849b-e34598d605e6/volumes" Oct 31 00:45:48.429199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:48.429183 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca827deb-869b-4423-a9ee-857c219e360d" path="/var/lib/kubelet/pods/ca827deb-869b-4423-a9ee-857c219e360d/volumes" Oct 31 00:45:48.658162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:48.658131 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:48.821137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:48.821104 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:49.900248 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:49.900211 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:51.032026 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.031986 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw"] Oct 31 00:45:51.041499 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.041439 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0faf990-rfjvw"] Oct 31 00:45:51.208794 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.208760 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f"] Oct 31 00:45:51.209075 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.209044 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" podUID="6fe078f0-9060-431d-8fe4-59ec4436c5d8" containerName="maas-api" containerID="cri-o://516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2" gracePeriod=30 Oct 31 00:45:51.311557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.311469 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:51.616523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.615017 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:51.651552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.651492 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-metrics-certs\") pod \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " Oct 31 00:45:51.651954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.651864 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls\") pod \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " Oct 31 00:45:51.652075 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.651959 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe078f0-9060-431d-8fe4-59ec4436c5d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe078f0-9060-431d-8fe4-59ec4436c5d8-openshift-service-ca\") pod \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " Oct 31 00:45:51.652075 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.652015 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe078f0-9060-431d-8fe4-59ec4436c5d8-kube-api-access-ghgjb\" (UniqueName: \"kubernetes.io/projected/6fe078f0-9060-431d-8fe4-59ec4436c5d8-kube-api-access-ghgjb\") pod \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\" (UID: \"6fe078f0-9060-431d-8fe4-59ec4436c5d8\") " Oct 31 00:45:51.652545 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.652415 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe078f0-9060-431d-8fe4-59ec4436c5d8-openshift-service-ca" pod "6fe078f0-9060-431d-8fe4-59ec4436c5d8" (UID: "6fe078f0-9060-431d-8fe4-59ec4436c5d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:51.653027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.652986 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe078f0-9060-431d-8fe4-59ec4436c5d8-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:51.655627 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.655564 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe078f0-9060-431d-8fe4-59ec4436c5d8-kube-api-access-ghgjb" pod "6fe078f0-9060-431d-8fe4-59ec4436c5d8" (UID: "6fe078f0-9060-431d-8fe4-59ec4436c5d8"). InnerVolumeSpecName "kube-api-access-ghgjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:51.661528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.661445 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-metrics-certs" pod "6fe078f0-9060-431d-8fe4-59ec4436c5d8" (UID: "6fe078f0-9060-431d-8fe4-59ec4436c5d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:51.661927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.661901 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls" pod "6fe078f0-9060-431d-8fe4-59ec4436c5d8" (UID: "6fe078f0-9060-431d-8fe4-59ec4436c5d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:51.756065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.756013 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:51.756065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.756051 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghgjb\" (UniqueName: \"kubernetes.io/projected/6fe078f0-9060-431d-8fe4-59ec4436c5d8-kube-api-access-ghgjb\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:51.756065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.756070 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fe078f0-9060-431d-8fe4-59ec4436c5d8-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:45:51.858249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:51.858152 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:52.020828 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.020789 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:52.031167 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.031131 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk"] Oct 31 00:45:52.036439 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.036408 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f598cd05-p2kvk"] Oct 31 00:45:52.072806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.072776 5978 generic.go:356] "Generic (PLEG): container finished" podID="6fe078f0-9060-431d-8fe4-59ec4436c5d8" containerID="516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2" exitCode=0 Oct 31 00:45:52.073008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.072829 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" Oct 31 00:45:52.073008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.072860 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" event={"ID":"6fe078f0-9060-431d-8fe4-59ec4436c5d8","Type":"ContainerDied","Data":"516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2"} Oct 31 00:45:52.073008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.072924 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f" event={"ID":"6fe078f0-9060-431d-8fe4-59ec4436c5d8","Type":"ContainerDied","Data":"e29cf160da46e86568f55848a5595c9c96d795ca43c336b61baea39ff91f96e6"} Oct 31 00:45:52.073008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.072955 5978 scope.go:122] "RemoveContainer" containerID="516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2" Oct 31 00:45:52.086970 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.086946 5978 scope.go:122] "RemoveContainer" containerID="516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2" Oct 31 00:45:52.087314 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:45:52.087283 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2\": container with ID starting with 516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2 not found: ID does not exist" containerID="516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2" Oct 31 00:45:52.087403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.087328 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2"} err="failed to get container status \"516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2\": rpc error: code = NotFound desc = could not find container \"516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2\": container with ID starting with 516ca1bb091c5159020004867bc8dddf7b01dc9fd0fda272d5367b7a215934e2 not found: ID does not exist" Oct 31 00:45:52.134542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.131949 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f"] Oct 31 00:45:52.146794 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.146752 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f598cd05-944f49478-ptb2f"] Oct 31 00:45:52.204824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.204787 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:52.357304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.357264 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:52.478829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.471906 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bdcf2a5-43ff-431b-9c67-352ebe4f5194" path="/var/lib/kubelet/pods/4bdcf2a5-43ff-431b-9c67-352ebe4f5194/volumes" Oct 31 00:45:52.478829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.472788 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe078f0-9060-431d-8fe4-59ec4436c5d8" path="/var/lib/kubelet/pods/6fe078f0-9060-431d-8fe4-59ec4436c5d8/volumes" Oct 31 00:45:52.478829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:52.473536 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da44a50-8b2e-4669-a550-f278371c4c6d" path="/var/lib/kubelet/pods/8da44a50-8b2e-4669-a550-f278371c4c6d/volumes" Oct 31 00:45:53.535093 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:53.535044 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:54.157373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:54.157328 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:54.319842 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:54.314136 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:56.522685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:56.522639 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:58.470647 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:58.470606 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:58.625191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:58.625150 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:58.730408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:58.730305 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp"] Oct 31 00:45:58.775851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:58.775816 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:58.796297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:58.796259 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658"] Oct 31 00:45:59.001880 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:59.001775 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:59.155265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:59.155231 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:59.480757 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:45:59.480725 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:00.893241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:00.893201 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:01.865356 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:01.865321 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:02.035249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:02.035211 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg"] Oct 31 00:46:02.046014 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:02.045972 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ba702f-7vvzg"] Oct 31 00:46:02.048561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:02.048516 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:02.429027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:02.428940 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875a4139-a4c8-4eae-81fb-961a0321257c" path="/var/lib/kubelet/pods/875a4139-a4c8-4eae-81fb-961a0321257c/volumes" Oct 31 00:46:03.183742 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:46:03.183705 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:46:03.480219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.480133 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc"] Oct 31 00:46:03.480961 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.480940 5978 cpu_manager.go:414] "containerMap: removing container" podUID="6fe078f0-9060-431d-8fe4-59ec4436c5d8" containerName="maas-api" Oct 31 00:46:03.480961 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.480957 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe078f0-9060-431d-8fe4-59ec4436c5d8" containerName="maas-api" Oct 31 00:46:03.481185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.481162 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe078f0-9060-431d-8fe4-59ec4436c5d8" containerName="maas-api" Oct 31 00:46:03.485312 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.485291 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.500875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.500846 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c2cd8283-serving-cert\"" Oct 31 00:46:03.516042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.516009 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc"] Oct 31 00:46:03.603231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.603193 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5878a827-0ffa-41d3-bbea-81a752b2d3fe-openshift-service-ca\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.603434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.603256 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgpj\" (UniqueName: \"kubernetes.io/projected/5878a827-0ffa-41d3-bbea-81a752b2d3fe-kube-api-access-8lgpj\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.603434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.603398 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.603700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.603677 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-metrics-certs\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.661007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.660974 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:03.705821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.705785 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5878a827-0ffa-41d3-bbea-81a752b2d3fe-openshift-service-ca\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.706000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.705842 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgpj\" (UniqueName: \"kubernetes.io/projected/5878a827-0ffa-41d3-bbea-81a752b2d3fe-kube-api-access-8lgpj\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.706000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.705887 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.706079 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.706019 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-metrics-certs\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.706283 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:46:03.706254 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-serving-cert: secret "maas-api-e2e-derive-c2cd8283-serving-cert" not found Oct 31 00:46:03.706562 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:46:03.706544 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls podName:5878a827-0ffa-41d3-bbea-81a752b2d3fe nodeName:}" failed. No retries permitted until 2026-10-31 00:46:04.206523526 +0000 UTC m=+1581.904637273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls") pod "maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" (UID: "5878a827-0ffa-41d3-bbea-81a752b2d3fe") : secret "maas-api-e2e-derive-c2cd8283-serving-cert" not found Oct 31 00:46:03.706769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.706742 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5878a827-0ffa-41d3-bbea-81a752b2d3fe-openshift-service-ca\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.712158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.712120 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-metrics-certs\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.728995 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.728206 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgpj\" (UniqueName: \"kubernetes.io/projected/5878a827-0ffa-41d3-bbea-81a752b2d3fe-kube-api-access-8lgpj\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:03.730861 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.730791 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" podUID="0534b4c0-79f4-47bd-9ff5-ec6b62da4777" containerName="payload-processing" containerID="cri-o://e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae" gracePeriod=25 Oct 31 00:46:03.797093 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.797050 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" podUID="8c8785f3-8a95-4ac7-8512-5df4123f2fda" containerName="payload-pre-processing" containerID="cri-o://84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e" gracePeriod=25 Oct 31 00:46:03.821567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.821529 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:03.978356 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:03.978263 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:04.099764 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.099728 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:46:04.178022 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.177989 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:46:04.216581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.216541 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-kube-api-access-nzpwk\" (UniqueName: \"kubernetes.io/projected/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-kube-api-access-nzpwk\") pod \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\" (UID: \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\") " Oct 31 00:46:04.217168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.217142 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-plugins-config-volume\") pod \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\" (UID: \"0534b4c0-79f4-47bd-9ff5-ec6b62da4777\") " Oct 31 00:46:04.217823 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.217792 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:04.223990 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.223953 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-plugins-config-volume" pod "0534b4c0-79f4-47bd-9ff5-ec6b62da4777" (UID: "0534b4c0-79f4-47bd-9ff5-ec6b62da4777"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:04.225113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.224838 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:04.228894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.228855 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls\") pod \"maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:04.232176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.232137 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-kube-api-access-nzpwk" pod "0534b4c0-79f4-47bd-9ff5-ec6b62da4777" (UID: "0534b4c0-79f4-47bd-9ff5-ec6b62da4777"). InnerVolumeSpecName "kube-api-access-nzpwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:04.326697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.326661 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8785f3-8a95-4ac7-8512-5df4123f2fda-kube-api-access-vf5fk\" (UniqueName: \"kubernetes.io/projected/8c8785f3-8a95-4ac7-8512-5df4123f2fda-kube-api-access-vf5fk\") pod \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\" (UID: \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\") " Oct 31 00:46:04.326847 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.326813 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8785f3-8a95-4ac7-8512-5df4123f2fda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8785f3-8a95-4ac7-8512-5df4123f2fda-plugins-config-volume\") pod \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\" (UID: \"8c8785f3-8a95-4ac7-8512-5df4123f2fda\") " Oct 31 00:46:04.327171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.327150 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8785f3-8a95-4ac7-8512-5df4123f2fda-plugins-config-volume" pod "8c8785f3-8a95-4ac7-8512-5df4123f2fda" (UID: "8c8785f3-8a95-4ac7-8512-5df4123f2fda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:04.327506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.327488 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8785f3-8a95-4ac7-8512-5df4123f2fda-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:04.327561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.327513 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzpwk\" (UniqueName: \"kubernetes.io/projected/0534b4c0-79f4-47bd-9ff5-ec6b62da4777-kube-api-access-nzpwk\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:04.328957 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.328936 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8785f3-8a95-4ac7-8512-5df4123f2fda-kube-api-access-vf5fk" pod "8c8785f3-8a95-4ac7-8512-5df4123f2fda" (UID: "8c8785f3-8a95-4ac7-8512-5df4123f2fda"). InnerVolumeSpecName "kube-api-access-vf5fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:04.355282 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.355247 5978 generic.go:356] "Generic (PLEG): container finished" podID="0534b4c0-79f4-47bd-9ff5-ec6b62da4777" containerID="e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae" exitCode=0 Oct 31 00:46:04.355509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.355303 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" Oct 31 00:46:04.355509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.355332 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" event={"ID":"0534b4c0-79f4-47bd-9ff5-ec6b62da4777","Type":"ContainerDied","Data":"e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae"} Oct 31 00:46:04.355509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.355386 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp" event={"ID":"0534b4c0-79f4-47bd-9ff5-ec6b62da4777","Type":"ContainerDied","Data":"a361517c8eec0a06039ca10872252fb9e180fb4e43a7c9834b0f335573defa11"} Oct 31 00:46:04.355509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.355409 5978 scope.go:122] "RemoveContainer" containerID="e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae" Oct 31 00:46:04.357057 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.357027 5978 generic.go:356] "Generic (PLEG): container finished" podID="8c8785f3-8a95-4ac7-8512-5df4123f2fda" containerID="84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e" exitCode=0 Oct 31 00:46:04.357257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.357068 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" event={"ID":"8c8785f3-8a95-4ac7-8512-5df4123f2fda","Type":"ContainerDied","Data":"84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e"} Oct 31 00:46:04.357257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.357074 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" Oct 31 00:46:04.357257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.357149 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658" event={"ID":"8c8785f3-8a95-4ac7-8512-5df4123f2fda","Type":"ContainerDied","Data":"ea4402e7375b97bbff6475be8bdd1d8bfa7c5d71dc687812b7ff84dae7fc0c28"} Oct 31 00:46:04.368032 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.368005 5978 scope.go:122] "RemoveContainer" containerID="e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae" Oct 31 00:46:04.368354 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:46:04.368331 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae\": container with ID starting with e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae not found: ID does not exist" containerID="e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae" Oct 31 00:46:04.368442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.368367 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae"} err="failed to get container status \"e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae\": rpc error: code = NotFound desc = could not find container \"e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae\": container with ID starting with e6cdcb34bf80c2798cc5a24e39bee3f5ac2524e7cb0c3009232ef3245405efae not found: ID does not exist" Oct 31 00:46:04.368442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.368390 5978 scope.go:122] "RemoveContainer" containerID="84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e" Oct 31 00:46:04.379373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.379348 5978 scope.go:122] "RemoveContainer" containerID="84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e" Oct 31 00:46:04.379762 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:46:04.379734 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e\": container with ID starting with 84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e not found: ID does not exist" containerID="84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e" Oct 31 00:46:04.379845 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.379774 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e"} err="failed to get container status \"84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e\": rpc error: code = NotFound desc = could not find container \"84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e\": container with ID starting with 84a50b69af08eb37cb4939e45d354d67b0f936dc0256b86c6d7c551601d9d32e not found: ID does not exist" Oct 31 00:46:04.401767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.401733 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:04.428620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.428591 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf5fk\" (UniqueName: \"kubernetes.io/projected/8c8785f3-8a95-4ac7-8512-5df4123f2fda-kube-api-access-vf5fk\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:04.429572 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.429551 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658"] Oct 31 00:46:04.429639 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.429589 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b08f569a-67cfcfcb95-vk658"] Oct 31 00:46:04.451550 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.451481 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp"] Oct 31 00:46:04.457155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.457119 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b08f569a-5bfc7f99cc-ttlzp"] Oct 31 00:46:04.731605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.731571 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc"] Oct 31 00:46:04.731891 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:46:04.731861 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5878a827_0ffa_41d3_bbea_81a752b2d3fe.slice/crio-97eb2707fafbdbd7e65949490d984c1be5921f064f44414d470cf540086a4cc5 WatchSource:0}: Error finding container 97eb2707fafbdbd7e65949490d984c1be5921f064f44414d470cf540086a4cc5: Status 404 returned error can't find the container with id 97eb2707fafbdbd7e65949490d984c1be5921f064f44414d470cf540086a4cc5 Oct 31 00:46:04.733649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:04.733628 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:46:05.373217 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:05.373172 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" event={"ID":"5878a827-0ffa-41d3-bbea-81a752b2d3fe","Type":"ContainerStarted","Data":"3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e"} Oct 31 00:46:05.373737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:05.373281 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" event={"ID":"5878a827-0ffa-41d3-bbea-81a752b2d3fe","Type":"ContainerStarted","Data":"97eb2707fafbdbd7e65949490d984c1be5921f064f44414d470cf540086a4cc5"} Oct 31 00:46:05.373737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:05.373435 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:05.413763 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:05.413690 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" podStartSLOduration=2.091279273 podStartE2EDuration="2.413671814s" totalImagesPullingTime="322.392541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:05.410329418 +0000 UTC m=+1583.108443184" watchObservedRunningTime="2026-10-31 00:46:05.413671814 +0000 UTC m=+1583.111785579" Oct 31 00:46:05.732267 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:05.732182 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:05.879145 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:05.879103 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:06.055619 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:06.055533 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9"] Oct 31 00:46:06.064176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:06.064119 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9461f01-nnmf9"] Oct 31 00:46:06.429862 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:06.429825 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0534b4c0-79f4-47bd-9ff5-ec6b62da4777" path="/var/lib/kubelet/pods/0534b4c0-79f4-47bd-9ff5-ec6b62da4777/volumes" Oct 31 00:46:06.430402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:06.430386 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8867d6c0-ae02-4c82-9aed-bd517ed149d0" path="/var/lib/kubelet/pods/8867d6c0-ae02-4c82-9aed-bd517ed149d0/volumes" Oct 31 00:46:06.430874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:06.430855 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8785f3-8a95-4ac7-8512-5df4123f2fda" path="/var/lib/kubelet/pods/8c8785f3-8a95-4ac7-8512-5df4123f2fda/volumes" Oct 31 00:46:06.761653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:06.761568 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:06.924339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:06.924077 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:07.068448 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:07.068412 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:07.614789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:07.614756 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:08.566447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:08.566411 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:08.714824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:08.714788 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:08.913539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:08.913415 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:09.068919 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:09.068882 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:10.914169 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:10.914134 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:11.419971 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:11.419936 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:12.208222 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:12.208184 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:12.705349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:12.705237 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:14.313851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:14.313811 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:15.039081 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:15.039038 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp"] Oct 31 00:46:15.048343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:15.048309 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995e74cb-2nrrp"] Oct 31 00:46:15.384070 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:15.384031 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:15.545535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:15.545493 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:15.869279 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:15.869247 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:16.019735 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:16.019705 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:16.373748 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:16.373713 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:16.460515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:16.458173 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae" path="/var/lib/kubelet/pods/3b9eec2f-ea9e-4edb-bc72-cbca1ac91eae/volumes" Oct 31 00:46:16.841902 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:16.841868 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:17.821285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:17.821254 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:19.031085 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:19.031041 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp"] Oct 31 00:46:19.041118 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:19.041089 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac57320-x8trp"] Oct 31 00:46:19.447750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:19.447712 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:19.589882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:19.589845 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:19.735983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:19.735894 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:20.428031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:20.427999 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c75715-9366-48ae-8d71-c1ea2cdc0484" path="/var/lib/kubelet/pods/50c75715-9366-48ae-8d71-c1ea2cdc0484/volumes" Oct 31 00:46:21.040618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:21.040582 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:21.181388 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:21.181348 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:21.936615 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:21.936582 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:22.810046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:22.809994 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:22.954179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:22.954139 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:23.942191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:23.942153 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:26.244867 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:26.244825 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:28.203023 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:28.202981 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:29.773364 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:29.773328 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:30.911316 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:30.911276 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:31.078190 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.078155 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:31.245944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.245854 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:31.523826 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.523721 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:31.940540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.940505 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp"] Oct 31 00:46:31.941337 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.941319 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8c8785f3-8a95-4ac7-8512-5df4123f2fda" containerName="payload-pre-processing" Oct 31 00:46:31.941337 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.941337 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8785f3-8a95-4ac7-8512-5df4123f2fda" containerName="payload-pre-processing" Oct 31 00:46:31.941502 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.941357 5978 cpu_manager.go:414] "containerMap: removing container" podUID="0534b4c0-79f4-47bd-9ff5-ec6b62da4777" containerName="payload-processing" Oct 31 00:46:31.941502 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.941362 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0534b4c0-79f4-47bd-9ff5-ec6b62da4777" containerName="payload-processing" Oct 31 00:46:31.941629 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.941520 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8785f3-8a95-4ac7-8512-5df4123f2fda" containerName="payload-pre-processing" Oct 31 00:46:31.941629 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.941530 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="0534b4c0-79f4-47bd-9ff5-ec6b62da4777" containerName="payload-processing" Oct 31 00:46:31.945415 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.945395 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:31.968024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.967997 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:46:31.968172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.968073 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:46:31.975416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:31.975383 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp"] Oct 31 00:46:32.027903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.027856 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bthsz\" (UniqueName: \"kubernetes.io/projected/b49c87f5-dcd1-4c71-824e-2d4d70290883-kube-api-access-bthsz\") pod \"maas-api-revoke-sub-1c128725-v7nzp\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.028108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.028057 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c128725-v7nzp\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.028180 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.028151 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c128725-v7nzp\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.129258 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.129224 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c128725-v7nzp\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.129443 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.129419 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bthsz\" (UniqueName: \"kubernetes.io/projected/b49c87f5-dcd1-4c71-824e-2d4d70290883-kube-api-access-bthsz\") pod \"maas-api-revoke-sub-1c128725-v7nzp\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.129580 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.129565 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c128725-v7nzp\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.129940 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.129913 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c128725-v7nzp\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.132024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.131988 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c128725-v7nzp\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.150836 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.150799 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bthsz\" (UniqueName: \"kubernetes.io/projected/b49c87f5-dcd1-4c71-824e-2d4d70290883-kube-api-access-bthsz\") pod \"maas-api-revoke-sub-1c128725-v7nzp\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.261872 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.261787 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:32.569315 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:46:32.569275 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49c87f5_dcd1_4c71_824e_2d4d70290883.slice/crio-f797617af251de0ce0936db6c56f8a99df40e3e3533350e7117016c9a2197697 WatchSource:0}: Error finding container f797617af251de0ce0936db6c56f8a99df40e3e3533350e7117016c9a2197697: Status 404 returned error can't find the container with id f797617af251de0ce0936db6c56f8a99df40e3e3533350e7117016c9a2197697 Oct 31 00:46:32.570099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.570072 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp"] Oct 31 00:46:32.708099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:32.708059 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" event={"ID":"b49c87f5-dcd1-4c71-824e-2d4d70290883","Type":"ContainerStarted","Data":"f797617af251de0ce0936db6c56f8a99df40e3e3533350e7117016c9a2197697"} Oct 31 00:46:33.185198 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:46:33.185112 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:46:33.380842 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:33.380802 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:33.719375 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:33.719343 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" event={"ID":"b49c87f5-dcd1-4c71-824e-2d4d70290883","Type":"ContainerStarted","Data":"ca9fdb304360757c8e045ef719a44db3f72f326a1ee294447e3b28e0fb669b46"} Oct 31 00:46:33.757601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:33.757542 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" podStartSLOduration=2.757479601 podStartE2EDuration="2.757479601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:33.756053673 +0000 UTC m=+1611.454167448" watchObservedRunningTime="2026-10-31 00:46:33.757479601 +0000 UTC m=+1611.455593367" Oct 31 00:46:34.731915 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:34.731880 5978 generic.go:356] "Generic (PLEG): container finished" podID="b49c87f5-dcd1-4c71-824e-2d4d70290883" containerID="ca9fdb304360757c8e045ef719a44db3f72f326a1ee294447e3b28e0fb669b46" exitCode=0 Oct 31 00:46:34.732334 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:34.731936 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" event={"ID":"b49c87f5-dcd1-4c71-824e-2d4d70290883","Type":"ContainerDied","Data":"ca9fdb304360757c8e045ef719a44db3f72f326a1ee294447e3b28e0fb669b46"} Oct 31 00:46:35.180705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:35.180659 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:35.968434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:35.968399 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:36.104437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.096095 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-openshift-ca\") pod \"b49c87f5-dcd1-4c71-824e-2d4d70290883\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " Oct 31 00:46:36.104437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.096180 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-xks-ca\") pod \"b49c87f5-dcd1-4c71-824e-2d4d70290883\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " Oct 31 00:46:36.104437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.096272 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49c87f5-dcd1-4c71-824e-2d4d70290883-kube-api-access-bthsz\" (UniqueName: \"kubernetes.io/projected/b49c87f5-dcd1-4c71-824e-2d4d70290883-kube-api-access-bthsz\") pod \"b49c87f5-dcd1-4c71-824e-2d4d70290883\" (UID: \"b49c87f5-dcd1-4c71-824e-2d4d70290883\") " Oct 31 00:46:36.104437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.096521 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-openshift-ca" pod "b49c87f5-dcd1-4c71-824e-2d4d70290883" (UID: "b49c87f5-dcd1-4c71-824e-2d4d70290883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:36.104437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.096906 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:36.104437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.098278 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-xks-ca" pod "b49c87f5-dcd1-4c71-824e-2d4d70290883" (UID: "b49c87f5-dcd1-4c71-824e-2d4d70290883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:36.104437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.098423 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49c87f5-dcd1-4c71-824e-2d4d70290883-kube-api-access-bthsz" pod "b49c87f5-dcd1-4c71-824e-2d4d70290883" (UID: "b49c87f5-dcd1-4c71-824e-2d4d70290883"). InnerVolumeSpecName "kube-api-access-bthsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:36.254282 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.198167 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c87f5-dcd1-4c71-824e-2d4d70290883-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:36.254282 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.198192 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bthsz\" (UniqueName: \"kubernetes.io/projected/b49c87f5-dcd1-4c71-824e-2d4d70290883-kube-api-access-bthsz\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:36.760212 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.760125 5978 generic.go:356] "Generic (PLEG): container finished" podID="194d8e50-d32e-492e-a9ad-66d15823d000" containerID="a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85" exitCode=0 Oct 31 00:46:36.760212 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.760194 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" event={"ID":"194d8e50-d32e-492e-a9ad-66d15823d000","Type":"ContainerDied","Data":"a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85"} Oct 31 00:46:36.760782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.760763 5978 scope.go:122] "RemoveContainer" containerID="a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85" Oct 31 00:46:36.761968 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.761951 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" Oct 31 00:46:36.762066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.762006 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp" event={"ID":"b49c87f5-dcd1-4c71-824e-2d4d70290883","Type":"ContainerDied","Data":"f797617af251de0ce0936db6c56f8a99df40e3e3533350e7117016c9a2197697"} Oct 31 00:46:36.762066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.762055 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f797617af251de0ce0936db6c56f8a99df40e3e3533350e7117016c9a2197697" Oct 31 00:46:36.762570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.762542 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:36.762634 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.762616 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:36.762690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:36.762654 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:37.147118 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:37.141682 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:37.736049 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:37.736013 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:37.775554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:37.775514 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" event={"ID":"194d8e50-d32e-492e-a9ad-66d15823d000","Type":"ContainerStarted","Data":"79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40"} Oct 31 00:46:37.903599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:37.903560 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:38.256960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:38.256926 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:46:38.258670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:38.258638 5978 patch_prober.go:29] interesting pod/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:46:38.258843 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:38.258692 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:46:39.004438 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:39.004407 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:39.257521 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:39.257409 5978 patch_prober.go:29] interesting pod/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:46:39.257521 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:39.257500 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:46:39.835530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:39.833568 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:46:40.256851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:40.256766 5978 patch_prober.go:29] interesting pod/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:46:40.256851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:40.256839 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:46:40.858820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:40.858750 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:41.065245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:41.065205 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:41.256730 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:41.256647 5978 patch_prober.go:29] interesting pod/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:46:41.256960 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:41.256935 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:46:42.256873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:42.256834 5978 patch_prober.go:29] interesting pod/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:46:42.257304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:42.256928 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:46:42.561132 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:42.561024 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:42.790181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:42.790142 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:42.945729 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:42.945691 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:43.091426 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.091388 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:43.257044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.256950 5978 patch_prober.go:29] interesting pod/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:46:43.257044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.257011 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:46:43.466420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.457986 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97"] Oct 31 00:46:43.468354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.468163 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b49c87f5-dcd1-4c71-824e-2d4d70290883" containerName="revoke-keys" Oct 31 00:46:43.469640 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.469617 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49c87f5-dcd1-4c71-824e-2d4d70290883" containerName="revoke-keys" Oct 31 00:46:43.470116 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.470095 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49c87f5-dcd1-4c71-824e-2d4d70290883" containerName="revoke-keys" Oct 31 00:46:43.475344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.475324 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.476978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.476947 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97"] Oct 31 00:46:43.494215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.494183 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-208e9f32-openshift-default-dockercfg-rs5gs\"" Oct 31 00:46:43.494605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.494574 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-podinfo\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.494818 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.494796 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.494935 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.494921 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-credential-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.494993 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.494957 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkl26\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-kube-api-access-gkl26\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.495139 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.495117 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-data\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.495317 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.495288 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-certs\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.495464 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.495339 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-envoy\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.495464 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.495381 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-token\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.495464 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.495417 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9171fc04-6c22-4ca2-a581-bbff5c77205a-istiod-ca-cert\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.570031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.569998 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:43.600162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.600127 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-certs\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.600902 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.600879 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-envoy\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.601444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.601425 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-token\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.602723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.600799 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-certs\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.604085 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.602689 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9171fc04-6c22-4ca2-a581-bbff5c77205a-istiod-ca-cert\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.604676 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.604653 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-podinfo\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.605074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.604612 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9171fc04-6c22-4ca2-a581-bbff5c77205a-istiod-ca-cert\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.605683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.605663 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.606578 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.606556 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.607728 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.606805 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-credential-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.607898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.607871 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkl26\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-kube-api-access-gkl26\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.608414 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.608393 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-data\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.609089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.608820 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-data\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.609228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.607631 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-credential-socket\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.610198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.610161 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-podinfo\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.637532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.637497 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-envoy\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.652494 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.651190 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkl26\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-kube-api-access-gkl26\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.662824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.662100 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-token\") pod \"e2e-mt-208e9f32-openshift-default-77466675fc-qqt97\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:43.759604 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.756071 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:43.800753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:43.796786 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:44.137206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.137114 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:44.219745 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:46:44.219704 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9171fc04_6c22_4ca2_a581_bbff5c77205a.slice/crio-57f5995018fa61e94977a3aed9fb3dfce823907a2e08467715429009a5a8ceab WatchSource:0}: Error finding container 57f5995018fa61e94977a3aed9fb3dfce823907a2e08467715429009a5a8ceab: Status 404 returned error can't find the container with id 57f5995018fa61e94977a3aed9fb3dfce823907a2e08467715429009a5a8ceab Oct 31 00:46:44.219945 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.219922 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97"] Oct 31 00:46:44.222773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.222731 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:44.222885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.222825 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:44.222885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.222868 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:44.261107 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.261079 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:46:44.262561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.262535 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:46:44.625123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.624845 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:44.873127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.873090 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" event={"ID":"9171fc04-6c22-4ca2-a581-bbff5c77205a","Type":"ContainerStarted","Data":"1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942"} Oct 31 00:46:44.873341 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.873150 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" event={"ID":"9171fc04-6c22-4ca2-a581-bbff5c77205a","Type":"ContainerStarted","Data":"57f5995018fa61e94977a3aed9fb3dfce823907a2e08467715429009a5a8ceab"} Oct 31 00:46:44.922408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:44.922290 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" podStartSLOduration=1.922275624 podStartE2EDuration="1.922275624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:44.919178383 +0000 UTC m=+1622.617292170" watchObservedRunningTime="2026-10-31 00:46:44.922275624 +0000 UTC m=+1622.620389388" Oct 31 00:46:45.124434 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.124389 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:45.203050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.202901 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc"] Oct 31 00:46:45.203246 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.203216 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" podUID="5878a827-0ffa-41d3-bbea-81a752b2d3fe" containerName="maas-api" containerID="cri-o://3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e" gracePeriod=30 Oct 31 00:46:45.295666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.295628 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:45.444228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.444190 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:45.610286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.610256 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:45.701037 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.700994 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-metrics-certs\") pod \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " Oct 31 00:46:45.701197 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.701102 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5878a827-0ffa-41d3-bbea-81a752b2d3fe-kube-api-access-8lgpj\" (UniqueName: \"kubernetes.io/projected/5878a827-0ffa-41d3-bbea-81a752b2d3fe-kube-api-access-8lgpj\") pod \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " Oct 31 00:46:45.701294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.701207 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5878a827-0ffa-41d3-bbea-81a752b2d3fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5878a827-0ffa-41d3-bbea-81a752b2d3fe-openshift-service-ca\") pod \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " Oct 31 00:46:45.703692 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.701502 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls\") pod \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\" (UID: \"5878a827-0ffa-41d3-bbea-81a752b2d3fe\") " Oct 31 00:46:45.703692 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.701804 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5878a827-0ffa-41d3-bbea-81a752b2d3fe-openshift-service-ca" pod "5878a827-0ffa-41d3-bbea-81a752b2d3fe" (UID: "5878a827-0ffa-41d3-bbea-81a752b2d3fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:45.703692 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.702584 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5878a827-0ffa-41d3-bbea-81a752b2d3fe-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:45.705014 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.704976 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls" pod "5878a827-0ffa-41d3-bbea-81a752b2d3fe" (UID: "5878a827-0ffa-41d3-bbea-81a752b2d3fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:45.705168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.705100 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-metrics-certs" pod "5878a827-0ffa-41d3-bbea-81a752b2d3fe" (UID: "5878a827-0ffa-41d3-bbea-81a752b2d3fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:45.705983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.705955 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5878a827-0ffa-41d3-bbea-81a752b2d3fe-kube-api-access-8lgpj" pod "5878a827-0ffa-41d3-bbea-81a752b2d3fe" (UID: "5878a827-0ffa-41d3-bbea-81a752b2d3fe"). InnerVolumeSpecName "kube-api-access-8lgpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:45.809479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.808614 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:45.809479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.808653 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lgpj\" (UniqueName: \"kubernetes.io/projected/5878a827-0ffa-41d3-bbea-81a752b2d3fe-kube-api-access-8lgpj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:45.809479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.808668 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5878a827-0ffa-41d3-bbea-81a752b2d3fe-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:45.814534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.810964 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:45.829842 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.829758 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:45.835876 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.835836 5978 scope.go:122] "RemoveContainer" containerID="6fa4399e690cf630e89046f7d594c681fe53c97d0c3b41c1b252b118bfffd575" Oct 31 00:46:45.851224 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.851198 5978 scope.go:122] "RemoveContainer" containerID="4007b116ae71f20dd7e4fa63067827726a48408539e05a691bfea2e0b8b1647a" Oct 31 00:46:45.866213 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.866180 5978 scope.go:122] "RemoveContainer" containerID="87b4e0f9694ac2e39db1cbebe68b0228980c52e032ec612b8aaf2bfe5f4ed201" Oct 31 00:46:45.887225 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.887184 5978 generic.go:356] "Generic (PLEG): container finished" podID="5878a827-0ffa-41d3-bbea-81a752b2d3fe" containerID="3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e" exitCode=0 Oct 31 00:46:45.888070 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.887282 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" Oct 31 00:46:45.888989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.887299 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" event={"ID":"5878a827-0ffa-41d3-bbea-81a752b2d3fe","Type":"ContainerDied","Data":"3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e"} Oct 31 00:46:45.888989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.888608 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc" event={"ID":"5878a827-0ffa-41d3-bbea-81a752b2d3fe","Type":"ContainerDied","Data":"97eb2707fafbdbd7e65949490d984c1be5921f064f44414d470cf540086a4cc5"} Oct 31 00:46:45.888989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.888515 5978 scope.go:122] "RemoveContainer" containerID="b27c3376fb28fd8adb1738b721e1a9dff9b48bba1491f465b4d0c3b4221ebb15" Oct 31 00:46:45.888989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.888730 5978 scope.go:122] "RemoveContainer" containerID="3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e" Oct 31 00:46:45.895800 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.895768 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:45.899687 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.899654 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:46:45.910927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.910900 5978 scope.go:122] "RemoveContainer" containerID="2480507d171a0ebb0055e1dc084b7e00664682b251c6fdc080f53d2c8b08c096" Oct 31 00:46:45.915514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.915428 5978 scope.go:122] "RemoveContainer" containerID="3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e" Oct 31 00:46:45.917408 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:46:45.917372 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e\": container with ID starting with 3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e not found: ID does not exist" containerID="3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e" Oct 31 00:46:45.917593 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.917427 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e"} err="failed to get container status \"3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e\": rpc error: code = NotFound desc = could not find container \"3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e\": container with ID starting with 3a979745fb5f850b911266a85a5850e1c6a047146da73bfefc0048bbdc30a21e not found: ID does not exist" Oct 31 00:46:45.943893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.943854 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc"] Oct 31 00:46:45.948042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:45.948004 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2cd8283-7bfbdd7497-wz4qc"] Oct 31 00:46:46.083374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:46.083256 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:46.291384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:46.291339 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:46.435805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:46.435717 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5878a827-0ffa-41d3-bbea-81a752b2d3fe" path="/var/lib/kubelet/pods/5878a827-0ffa-41d3-bbea-81a752b2d3fe/volumes" Oct 31 00:46:46.437514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:46.437479 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:46.481069 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:46.481016 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms"] Oct 31 00:46:46.482669 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:46.481431 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" containerID="cri-o://79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40" gracePeriod=30 Oct 31 00:46:46.597541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:46.597496 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:47.841100 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:47.841064 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:48.001852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:48.001807 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.026800 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.026765 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.249830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.249776 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.829167 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.829126 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.968579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.968541 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54"] Oct 31 00:46:49.969391 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.969376 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5878a827-0ffa-41d3-bbea-81a752b2d3fe" containerName="maas-api" Oct 31 00:46:49.969442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.969392 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5878a827-0ffa-41d3-bbea-81a752b2d3fe" containerName="maas-api" Oct 31 00:46:49.969553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.969542 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5878a827-0ffa-41d3-bbea-81a752b2d3fe" containerName="maas-api" Oct 31 00:46:49.973953 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.973934 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:46:49.983079 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.983036 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.989154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.989128 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcmj5\" (UniqueName: \"kubernetes.io/projected/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-kube-api-access-rcmj5\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54\" (UID: \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:46:49.989331 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.989315 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54\" (UID: \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:46:49.998263 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.998234 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-208e9f32-dockercfg-b8pn6\"" Oct 31 00:46:49.998263 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:49.998232 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-208e9f32\"" Oct 31 00:46:50.005290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.005262 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54"] Oct 31 00:46:50.106541 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.099756 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg"] Oct 31 00:46:50.110966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.110688 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmj5\" (UniqueName: \"kubernetes.io/projected/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-kube-api-access-rcmj5\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54\" (UID: \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:46:50.117582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.117345 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54\" (UID: \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:46:50.121027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.119261 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54\" (UID: \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:46:50.157555 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.150597 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcmj5\" (UniqueName: \"kubernetes.io/projected/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-kube-api-access-rcmj5\") pod \"payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54\" (UID: \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:46:50.157555 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.155223 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg"] Oct 31 00:46:50.157555 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.155366 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:46:50.221531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.219803 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-plugins-config-volume\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg\" (UID: \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:46:50.221531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.219980 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45tdr\" (UniqueName: \"kubernetes.io/projected/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-kube-api-access-45tdr\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg\" (UID: \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:46:50.288986 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.288947 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:46:50.321822 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.321777 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45tdr\" (UniqueName: \"kubernetes.io/projected/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-kube-api-access-45tdr\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg\" (UID: \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:46:50.322027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.322013 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-plugins-config-volume\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg\" (UID: \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:46:50.322988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.322958 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-plugins-config-volume\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg\" (UID: \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:46:50.348280 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.348244 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45tdr\" (UniqueName: \"kubernetes.io/projected/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-kube-api-access-45tdr\") pod \"payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg\" (UID: \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\") " pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:46:50.486926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.486732 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:50.499335 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.498035 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:46:50.635443 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.635411 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54"] Oct 31 00:46:50.874983 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:46:50.874905 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740368e2_d7f8_41ec_ac01_3e5b9b2fa707.slice/crio-36ade3d836ad385aae0e3cb9bd4ce1f3442eec9a8df142831c0bb06ad122664f WatchSource:0}: Error finding container 36ade3d836ad385aae0e3cb9bd4ce1f3442eec9a8df142831c0bb06ad122664f: Status 404 returned error can't find the container with id 36ade3d836ad385aae0e3cb9bd4ce1f3442eec9a8df142831c0bb06ad122664f Oct 31 00:46:50.877225 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.877197 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg"] Oct 31 00:46:50.957927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.957888 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" event={"ID":"be61c2cf-6d4a-40ab-a95b-e4413a1394a7","Type":"ContainerStarted","Data":"3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035"} Oct 31 00:46:50.958124 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.957957 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" event={"ID":"be61c2cf-6d4a-40ab-a95b-e4413a1394a7","Type":"ContainerStarted","Data":"3b420ea494a91eb741c8f6a9c4b1d178cb20255a4be0868d1bc55802689368cb"} Oct 31 00:46:50.958369 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.958346 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:46:50.960089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.960060 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" event={"ID":"740368e2-d7f8-41ec-ac01-3e5b9b2fa707","Type":"ContainerStarted","Data":"91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e"} Oct 31 00:46:50.960196 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.960117 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:46:50.960196 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.960142 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" event={"ID":"740368e2-d7f8-41ec-ac01-3e5b9b2fa707","Type":"ContainerStarted","Data":"36ade3d836ad385aae0e3cb9bd4ce1f3442eec9a8df142831c0bb06ad122664f"} Oct 31 00:46:50.999383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:50.999315 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" podStartSLOduration=1.999292934 podStartE2EDuration="1.999292934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:50.994966642 +0000 UTC m=+1628.693080408" watchObservedRunningTime="2026-10-31 00:46:50.999292934 +0000 UTC m=+1628.697406700" Oct 31 00:46:51.034698 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.034634 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" podStartSLOduration=1.034618676 podStartE2EDuration="1.034618676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:51.030687957 +0000 UTC m=+1628.728801723" watchObservedRunningTime="2026-10-31 00:46:51.034618676 +0000 UTC m=+1628.732732441" Oct 31 00:46:51.156603 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.156310 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:51.424564 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.424475 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:51.574467 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.574418 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:51.816184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.816155 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:46:51.874922 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.874886 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-data\") pod \"194d8e50-d32e-492e-a9ad-66d15823d000\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " Oct 31 00:46:51.875128 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.874986 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-kube-api-access-gtnj5\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-kube-api-access-gtnj5\") pod \"194d8e50-d32e-492e-a9ad-66d15823d000\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " Oct 31 00:46:51.875128 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875032 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-credential-socket\") pod \"194d8e50-d32e-492e-a9ad-66d15823d000\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " Oct 31 00:46:51.875128 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875093 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/194d8e50-d32e-492e-a9ad-66d15823d000-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194d8e50-d32e-492e-a9ad-66d15823d000-istio-podinfo\") pod \"194d8e50-d32e-492e-a9ad-66d15823d000\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " Oct 31 00:46:51.875287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875166 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-certs\") pod \"194d8e50-d32e-492e-a9ad-66d15823d000\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " Oct 31 00:46:51.875287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875208 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194d8e50-d32e-492e-a9ad-66d15823d000-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194d8e50-d32e-492e-a9ad-66d15823d000-istiod-ca-cert\") pod \"194d8e50-d32e-492e-a9ad-66d15823d000\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " Oct 31 00:46:51.875287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875220 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-data" pod "194d8e50-d32e-492e-a9ad-66d15823d000" (UID: "194d8e50-d32e-492e-a9ad-66d15823d000"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:51.875442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875289 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-socket\") pod \"194d8e50-d32e-492e-a9ad-66d15823d000\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " Oct 31 00:46:51.875442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875344 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-istio-token\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-istio-token\") pod \"194d8e50-d32e-492e-a9ad-66d15823d000\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " Oct 31 00:46:51.875442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875432 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-credential-socket" pod "194d8e50-d32e-492e-a9ad-66d15823d000" (UID: "194d8e50-d32e-492e-a9ad-66d15823d000"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:51.875616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875585 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-envoy\") pod \"194d8e50-d32e-492e-a9ad-66d15823d000\" (UID: \"194d8e50-d32e-492e-a9ad-66d15823d000\") " Oct 31 00:46:51.876261 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.875585 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-certs" pod "194d8e50-d32e-492e-a9ad-66d15823d000" (UID: "194d8e50-d32e-492e-a9ad-66d15823d000"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:51.876261 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.876126 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-socket" pod "194d8e50-d32e-492e-a9ad-66d15823d000" (UID: "194d8e50-d32e-492e-a9ad-66d15823d000"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:51.876514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.876490 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194d8e50-d32e-492e-a9ad-66d15823d000-istiod-ca-cert" pod "194d8e50-d32e-492e-a9ad-66d15823d000" (UID: "194d8e50-d32e-492e-a9ad-66d15823d000"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:51.877172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.877061 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:51.877172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.877090 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:51.877172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.877103 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194d8e50-d32e-492e-a9ad-66d15823d000-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:51.877172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.877118 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:51.877172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.877132 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:51.878566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.878537 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-envoy" pod "194d8e50-d32e-492e-a9ad-66d15823d000" (UID: "194d8e50-d32e-492e-a9ad-66d15823d000"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:51.878671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.878604 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-kube-api-access-gtnj5" pod "194d8e50-d32e-492e-a9ad-66d15823d000" (UID: "194d8e50-d32e-492e-a9ad-66d15823d000"). InnerVolumeSpecName "kube-api-access-gtnj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:51.878740 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.878665 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-istio-token" pod "194d8e50-d32e-492e-a9ad-66d15823d000" (UID: "194d8e50-d32e-492e-a9ad-66d15823d000"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:51.878740 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.878715 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/194d8e50-d32e-492e-a9ad-66d15823d000-istio-podinfo" pod "194d8e50-d32e-492e-a9ad-66d15823d000" (UID: "194d8e50-d32e-492e-a9ad-66d15823d000"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:46:51.971431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.971334 5978 generic.go:356] "Generic (PLEG): container finished" podID="194d8e50-d32e-492e-a9ad-66d15823d000" containerID="79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40" exitCode=0 Oct 31 00:46:51.971431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.971399 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" Oct 31 00:46:51.971431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.971419 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" event={"ID":"194d8e50-d32e-492e-a9ad-66d15823d000","Type":"ContainerDied","Data":"79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40"} Oct 31 00:46:51.971712 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.971492 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms" event={"ID":"194d8e50-d32e-492e-a9ad-66d15823d000","Type":"ContainerDied","Data":"eb15ef408959f210875d4e9bc21f86b7735f3aa29b7526078e3710bb4376e063"} Oct 31 00:46:51.971712 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.971516 5978 scope.go:122] "RemoveContainer" containerID="79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40" Oct 31 00:46:51.979000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.978969 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:51.979000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.979001 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194d8e50-d32e-492e-a9ad-66d15823d000-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:51.979168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.979017 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtnj5\" (UniqueName: \"kubernetes.io/projected/194d8e50-d32e-492e-a9ad-66d15823d000-kube-api-access-gtnj5\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:51.979168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.979027 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194d8e50-d32e-492e-a9ad-66d15823d000-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:46:51.984082 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.984055 5978 scope.go:122] "RemoveContainer" containerID="a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85" Oct 31 00:46:51.997234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.997199 5978 scope.go:122] "RemoveContainer" containerID="79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40" Oct 31 00:46:51.997660 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:46:51.997632 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40\": container with ID starting with 79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40 not found: ID does not exist" containerID="79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40" Oct 31 00:46:51.997782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.997674 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40"} err="failed to get container status \"79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40\": rpc error: code = NotFound desc = could not find container \"79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40\": container with ID starting with 79251f46513d36d3eba311efe8f55df7a9f35ade6552b43f2aa807f8a85cef40 not found: ID does not exist" Oct 31 00:46:51.997782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.997700 5978 scope.go:122] "RemoveContainer" containerID="a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85" Oct 31 00:46:51.998025 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:46:51.997992 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85\": container with ID starting with a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85 not found: ID does not exist" containerID="a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85" Oct 31 00:46:51.998131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:51.998035 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85"} err="failed to get container status \"a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85\": rpc error: code = NotFound desc = could not find container \"a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85\": container with ID starting with a2b3d9d612d1bbfd05f2b2169f0548ef9199934042bcd58ddfbabbe2f9437b85 not found: ID does not exist" Oct 31 00:46:52.021859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:52.021820 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms"] Oct 31 00:46:52.027131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:52.027097 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c2cd8283-openshift-default-59f64f5949-ftzms"] Oct 31 00:46:52.429978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:52.429944 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" path="/var/lib/kubelet/pods/194d8e50-d32e-492e-a9ad-66d15823d000/volumes" Oct 31 00:46:52.941910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:52.941870 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:53.071640 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.071600 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25"] Oct 31 00:46:53.072426 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.072411 5978 cpu_manager.go:414] "containerMap: removing container" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" Oct 31 00:46:53.072488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.072428 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" Oct 31 00:46:53.072638 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.072627 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" Oct 31 00:46:53.072682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.072645 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" Oct 31 00:46:53.076429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.076409 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:46:53.098493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.098427 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25"] Oct 31 00:46:53.101901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.101872 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f2bdf3\"" Oct 31 00:46:53.101901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.101891 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f2bdf3-dockercfg-9zfsq\"" Oct 31 00:46:53.105552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.105522 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:53.167522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.163151 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf"] Oct 31 00:46:53.167522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.164690 5978 cpu_manager.go:414] "containerMap: removing container" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" Oct 31 00:46:53.167522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.164718 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194d8e50-d32e-492e-a9ad-66d15823d000" containerName="istio-proxy" Oct 31 00:46:53.177553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.177508 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:46:53.270820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.246091 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85f151b-d6cf-4189-839b-1c55e2980350-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25\" (UID: \"b85f151b-d6cf-4189-839b-1c55e2980350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:46:53.270820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.246540 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wh7l\" (UniqueName: \"kubernetes.io/projected/b85f151b-d6cf-4189-839b-1c55e2980350-kube-api-access-8wh7l\") pod \"payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25\" (UID: \"b85f151b-d6cf-4189-839b-1c55e2980350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:46:53.363147 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.363103 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wh7l\" (UniqueName: \"kubernetes.io/projected/b85f151b-d6cf-4189-839b-1c55e2980350-kube-api-access-8wh7l\") pod \"payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25\" (UID: \"b85f151b-d6cf-4189-839b-1c55e2980350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:46:53.363372 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.363316 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85f151b-d6cf-4189-839b-1c55e2980350-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25\" (UID: \"b85f151b-d6cf-4189-839b-1c55e2980350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:46:53.363445 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.363371 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4csm\" (UniqueName: \"kubernetes.io/projected/a99d5479-9fc7-4cbd-a242-9f2c33cead38-kube-api-access-h4csm\") pod \"payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf\" (UID: \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:46:53.363531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.363503 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d5479-9fc7-4cbd-a242-9f2c33cead38-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf\" (UID: \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:46:53.369921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.369860 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85f151b-d6cf-4189-839b-1c55e2980350-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25\" (UID: \"b85f151b-d6cf-4189-839b-1c55e2980350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:46:53.379980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.379952 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf"] Oct 31 00:46:53.392364 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.392315 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wh7l\" (UniqueName: \"kubernetes.io/projected/b85f151b-d6cf-4189-839b-1c55e2980350-kube-api-access-8wh7l\") pod \"payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25\" (UID: \"b85f151b-d6cf-4189-839b-1c55e2980350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:46:53.464842 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.464804 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4csm\" (UniqueName: \"kubernetes.io/projected/a99d5479-9fc7-4cbd-a242-9f2c33cead38-kube-api-access-h4csm\") pod \"payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf\" (UID: \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:46:53.465319 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.464914 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d5479-9fc7-4cbd-a242-9f2c33cead38-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf\" (UID: \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:46:53.465700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.465679 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d5479-9fc7-4cbd-a242-9f2c33cead38-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf\" (UID: \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:46:53.486516 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.486444 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4csm\" (UniqueName: \"kubernetes.io/projected/a99d5479-9fc7-4cbd-a242-9f2c33cead38-kube-api-access-h4csm\") pod \"payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf\" (UID: \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:46:53.502266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.502238 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:53.691417 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.691366 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:46:53.697204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:53.697172 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:46:54.034266 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:46:54.034236 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85f151b_d6cf_4189_839b_1c55e2980350.slice/crio-8655e3b0d32e7ca9ff9d07c533fa3aa0ccf0b4823c4ad2130c88e7d481be421b WatchSource:0}: Error finding container 8655e3b0d32e7ca9ff9d07c533fa3aa0ccf0b4823c4ad2130c88e7d481be421b: Status 404 returned error can't find the container with id 8655e3b0d32e7ca9ff9d07c533fa3aa0ccf0b4823c4ad2130c88e7d481be421b Oct 31 00:46:54.036591 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.036555 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25"] Oct 31 00:46:54.054250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.054211 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf"] Oct 31 00:46:54.056696 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:46:54.056657 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99d5479_9fc7_4cbd_a242_9f2c33cead38.slice/crio-7335d7bf44ee612b230f74f5b7033439f5d5d08846cfa5f8babd5b4dac66c3a5 WatchSource:0}: Error finding container 7335d7bf44ee612b230f74f5b7033439f5d5d08846cfa5f8babd5b4dac66c3a5: Status 404 returned error can't find the container with id 7335d7bf44ee612b230f74f5b7033439f5d5d08846cfa5f8babd5b4dac66c3a5 Oct 31 00:46:54.268272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.268236 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf"] Oct 31 00:46:54.272470 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.272427 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.286086 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.286057 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-36af31-openshift-default-dockercfg-vv85m\"" Oct 31 00:46:54.306727 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.306690 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf"] Oct 31 00:46:54.381382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.381344 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/395c04b2-9440-4612-8711-5566faf2c7c3-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.381619 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.381401 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k442\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-kube-api-access-4k442\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.381619 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.381587 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/395c04b2-9440-4612-8711-5566faf2c7c3-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.381730 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.381706 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.381904 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.381885 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.381959 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.381947 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.382073 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.382057 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.382112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.382101 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.382191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.382175 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.483924 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.483891 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/395c04b2-9440-4612-8711-5566faf2c7c3-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.483941 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k442\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-kube-api-access-4k442\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484005 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/395c04b2-9440-4612-8711-5566faf2c7c3-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484061 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484149 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484183 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484314 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484397 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484365 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484417 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484594 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-credential-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484619 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-data\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484661 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-certs\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484706 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/395c04b2-9440-4612-8711-5566faf2c7c3-istiod-ca-cert\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.484927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.484793 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-socket\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.486965 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.486938 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-envoy\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.487087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.486983 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/395c04b2-9440-4612-8711-5566faf2c7c3-istio-podinfo\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.505562 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.505528 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-istio-token\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.505729 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.505674 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k442\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-kube-api-access-4k442\") pod \"e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.587495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.587445 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:54.918223 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.917990 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf"] Oct 31 00:46:54.919902 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.919854 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:54.920016 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.919951 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:54.920016 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:54.919995 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:55.005308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.005268 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" event={"ID":"395c04b2-9440-4612-8711-5566faf2c7c3","Type":"ContainerStarted","Data":"61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50"} Oct 31 00:46:55.005546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.005339 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" event={"ID":"395c04b2-9440-4612-8711-5566faf2c7c3","Type":"ContainerStarted","Data":"f500d1428d76fd6f580bbbd8371928bc455d9f6f60284a028dcab8fe5ef69df1"} Oct 31 00:46:55.007550 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.007518 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" event={"ID":"a99d5479-9fc7-4cbd-a242-9f2c33cead38","Type":"ContainerStarted","Data":"3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7"} Oct 31 00:46:55.007697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.007583 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:46:55.007697 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.007604 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" event={"ID":"a99d5479-9fc7-4cbd-a242-9f2c33cead38","Type":"ContainerStarted","Data":"7335d7bf44ee612b230f74f5b7033439f5d5d08846cfa5f8babd5b4dac66c3a5"} Oct 31 00:46:55.009482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.009433 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" event={"ID":"b85f151b-d6cf-4189-839b-1c55e2980350","Type":"ContainerStarted","Data":"ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1"} Oct 31 00:46:55.009601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.009504 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" event={"ID":"b85f151b-d6cf-4189-839b-1c55e2980350","Type":"ContainerStarted","Data":"8655e3b0d32e7ca9ff9d07c533fa3aa0ccf0b4823c4ad2130c88e7d481be421b"} Oct 31 00:46:55.009601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.009551 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:46:55.056062 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.055990 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" podStartSLOduration=1.055967096 podStartE2EDuration="1.055967096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:55.046984131 +0000 UTC m=+1632.745097921" watchObservedRunningTime="2026-10-31 00:46:55.055967096 +0000 UTC m=+1632.754080862" Oct 31 00:46:55.087513 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.087422 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" podStartSLOduration=2.087400708 podStartE2EDuration="2.087400708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:55.082938276 +0000 UTC m=+1632.781052041" watchObservedRunningTime="2026-10-31 00:46:55.087400708 +0000 UTC m=+1632.785514476" Oct 31 00:46:55.124472 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.124391 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" podStartSLOduration=2.124368661 podStartE2EDuration="2.124368661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:55.120796305 +0000 UTC m=+1632.818910073" watchObservedRunningTime="2026-10-31 00:46:55.124368661 +0000 UTC m=+1632.822482458" Oct 31 00:46:55.146372 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.146333 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:55.292685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.292562 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:55.450250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.450217 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:55.593026 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.588008 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:46:55.593026 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.590380 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:46:55.593026 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:55.590444 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 00:46:56.588850 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:56.588810 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:46:56.589044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:56.588868 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 00:46:57.353363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:57.353235 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:57.588875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:57.588837 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:46:57.589058 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:57.588900 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 00:46:58.588714 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:58.588681 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:46:58.589154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:58.588738 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 00:46:59.140479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:59.137250 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:59.588191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:59.588146 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:46:59.588409 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:46:59.588211 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 00:47:00.588582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:00.588544 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:47:00.589018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:00.588618 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 00:47:00.895922 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:00.895835 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:01.594386 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:01.594344 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:47:01.594948 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:01.594929 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:47:01.595963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:01.595940 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 00:47:01.979563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:01.976859 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:47:01.979563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:01.978331 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:47:02.882553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:02.882502 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:03.274554 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:47:03.262147 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:47:06.059032 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:06.059002 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 00:47:06.059605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:06.059582 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 00:47:06.465956 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:06.465917 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:06.737020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:06.736893 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:07.211901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:07.211858 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:07.722149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:07.722106 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:08.052964 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:08.052858 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:09.034064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:09.034028 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c"] Oct 31 00:47:09.043671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:09.043627 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036ab1d2-ml67c"] Oct 31 00:47:09.207708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:09.207667 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:09.398593 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:09.398494 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:09.544820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:09.544783 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:09.942444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:09.942412 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:10.132958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:10.132918 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:10.428878 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:10.428843 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547d423a-2bbb-4909-9a26-90a2338a8560" path="/var/lib/kubelet/pods/547d423a-2bbb-4909-9a26-90a2338a8560/volumes" Oct 31 00:47:11.464013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:11.463977 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:11.664817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:11.664782 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:11.836475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:11.836208 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:13.224148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:13.224093 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:13.509353 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:13.509252 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:14.205814 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:14.205773 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:15.732702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:15.732668 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl"] Oct 31 00:47:15.737709 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:15.737679 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:15.760654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:15.760604 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f2bdf3-serving-cert\"" Oct 31 00:47:15.770630 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:15.770594 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl"] Oct 31 00:47:15.881948 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:15.881916 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:15.919894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:15.919859 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:15.920079 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:15.919981 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:15.920179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:15.920160 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3bf388-5757-464e-a5a9-9d7378c295c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:15.920353 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:15.920336 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4jr\" (UniqueName: \"kubernetes.io/projected/8b3bf388-5757-464e-a5a9-9d7378c295c0-kube-api-access-qn4jr\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.021556 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.021418 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.021792 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.021707 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3bf388-5757-464e-a5a9-9d7378c295c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.021955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.021938 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4jr\" (UniqueName: \"kubernetes.io/projected/8b3bf388-5757-464e-a5a9-9d7378c295c0-kube-api-access-qn4jr\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.022027 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.021996 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.022568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.022545 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3bf388-5757-464e-a5a9-9d7378c295c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.024065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.024041 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.024438 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.024416 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.026148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.026125 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:16.044956 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.044922 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4jr\" (UniqueName: \"kubernetes.io/projected/8b3bf388-5757-464e-a5a9-9d7378c295c0-kube-api-access-qn4jr\") pod \"maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.054737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.054708 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:16.175671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.175640 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:16.373241 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:47:16.373210 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3bf388_5757_464e_a5a9_9d7378c295c0.slice/crio-55bed8881fe8756e82e97b4b40326dad7e33d056daea1f655594193d8e343563 WatchSource:0}: Error finding container 55bed8881fe8756e82e97b4b40326dad7e33d056daea1f655594193d8e343563: Status 404 returned error can't find the container with id 55bed8881fe8756e82e97b4b40326dad7e33d056daea1f655594193d8e343563 Oct 31 00:47:16.374743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.374716 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl"] Oct 31 00:47:16.387900 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:16.387874 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" event={"ID":"8b3bf388-5757-464e-a5a9-9d7378c295c0","Type":"ContainerStarted","Data":"55bed8881fe8756e82e97b4b40326dad7e33d056daea1f655594193d8e343563"} Oct 31 00:47:17.405800 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:17.405757 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" event={"ID":"8b3bf388-5757-464e-a5a9-9d7378c295c0","Type":"ContainerStarted","Data":"c3d896f526c2d197e7e6af1de3014c20b886c59e339c862244a7c89e514148b1"} Oct 31 00:47:17.406302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:17.405867 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:17.449571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:17.449518 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" podStartSLOduration=2.140801015 podStartE2EDuration="2.449503147s" totalImagesPullingTime="308.702132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:17.443528954 +0000 UTC m=+1655.141642722" watchObservedRunningTime="2026-10-31 00:47:17.449503147 +0000 UTC m=+1655.147616933" Oct 31 00:47:17.895500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:17.895152 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:19.749554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:19.749504 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:21.448802 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:21.448753 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:23.128415 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:23.128372 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:23.421999 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:23.421927 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 00:47:24.824809 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:24.824771 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:25.031877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:25.031841 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz"] Oct 31 00:47:25.041178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:25.041145 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5594c2f1-ckmqz"] Oct 31 00:47:26.427262 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:26.427215 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b752a492-f6d2-4854-aa00-0eb678e3f3bd" path="/var/lib/kubelet/pods/b752a492-f6d2-4854-aa00-0eb678e3f3bd/volumes" Oct 31 00:47:26.499138 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:26.499102 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:28.058217 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:28.058179 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:29.716383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:29.716346 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:31.308739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:31.308703 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:33.048160 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.048122 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7"] Oct 31 00:47:33.053555 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.053532 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.078184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.078148 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-3c90d8-serving-cert\"" Oct 31 00:47:33.084567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.084527 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7"] Oct 31 00:47:33.141438 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.141396 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.141887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.141856 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.142031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.141947 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7h5c\" (UniqueName: \"kubernetes.io/projected/52036be9-339e-4bb6-a5c1-30f41c943ff5-kube-api-access-g7h5c\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.142108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.142059 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036be9-339e-4bb6-a5c1-30f41c943ff5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.187619 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:47:33.187578 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:47:33.215554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.215513 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:33.243963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.243926 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.244169 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.243997 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7h5c\" (UniqueName: \"kubernetes.io/projected/52036be9-339e-4bb6-a5c1-30f41c943ff5-kube-api-access-g7h5c\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.244169 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.244076 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036be9-339e-4bb6-a5c1-30f41c943ff5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.244299 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.244175 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.244299 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:47:33.244080 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-serving-cert: secret "maas-api-e2e-shared-b-w0-3c90d8-serving-cert" not found Oct 31 00:47:33.244299 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:47:33.244281 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls podName:52036be9-339e-4bb6-a5c1-30f41c943ff5 nodeName:}" failed. No retries permitted until 2026-10-31 00:47:33.744260557 +0000 UTC m=+1671.442374300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls") pod "maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" (UID: "52036be9-339e-4bb6-a5c1-30f41c943ff5") : secret "maas-api-e2e-shared-b-w0-3c90d8-serving-cert" not found Oct 31 00:47:33.244808 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.244784 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036be9-339e-4bb6-a5c1-30f41c943ff5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.246660 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.246639 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.268249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.268212 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7h5c\" (UniqueName: \"kubernetes.io/projected/52036be9-339e-4bb6-a5c1-30f41c943ff5-kube-api-access-g7h5c\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.362618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.362582 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:33.509331 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.509288 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:33.754119 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.754022 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.756721 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.756690 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:33.970104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:33.970063 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:34.287320 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:47:34.287280 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52036be9_339e_4bb6_a5c1_30f41c943ff5.slice/crio-1a20a4cf361e4b751598d1f9559409182af04aebe12b1e19760bc692538612a0 WatchSource:0}: Error finding container 1a20a4cf361e4b751598d1f9559409182af04aebe12b1e19760bc692538612a0: Status 404 returned error can't find the container with id 1a20a4cf361e4b751598d1f9559409182af04aebe12b1e19760bc692538612a0 Oct 31 00:47:34.287979 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:34.287951 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7"] Oct 31 00:47:34.602527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:34.602440 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" event={"ID":"52036be9-339e-4bb6-a5c1-30f41c943ff5","Type":"ContainerStarted","Data":"1a20a4cf361e4b751598d1f9559409182af04aebe12b1e19760bc692538612a0"} Oct 31 00:47:35.380367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:35.379121 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:35.613736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:35.613694 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" event={"ID":"52036be9-339e-4bb6-a5c1-30f41c943ff5","Type":"ContainerStarted","Data":"3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a"} Oct 31 00:47:35.613899 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:35.613782 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:35.658579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:35.658472 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" podStartSLOduration=2.307977269 podStartE2EDuration="2.658443833s" totalImagesPullingTime="350.466564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:35.653657842 +0000 UTC m=+1673.351771609" watchObservedRunningTime="2026-10-31 00:47:35.658443833 +0000 UTC m=+1673.356557595" Oct 31 00:47:36.264044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.262280 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:36.660414 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.660378 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:36.726573 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.726535 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf"] Oct 31 00:47:36.731415 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.731389 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.747396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.747338 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-bcdb4e-openshift-default-dockercfg-s86d8\"" Oct 31 00:47:36.780933 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.772737 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf"] Oct 31 00:47:36.843786 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.843750 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:36.870548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.865224 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.870548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.865338 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-podinfo\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.870548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.865510 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-token\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.870548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.865561 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsf6c\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-kube-api-access-nsf6c\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.870548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.865673 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istiod-ca-cert\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.870548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.865749 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-data\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.870548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.865782 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-credential-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.870548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.865861 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-certs\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.870548 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.865922 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-envoy\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.977531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.976792 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istiod-ca-cert\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.977531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.976888 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-data\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.977531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.976929 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-credential-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.977531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.976997 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-certs\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.977531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.977050 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-envoy\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.977531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.977273 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.977531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.977320 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-podinfo\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.977531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.977425 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-token\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:36.977531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.977482 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsf6c\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-kube-api-access-nsf6c\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.007477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.987255 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istiod-ca-cert\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.007477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.987584 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.007477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.999009 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-data\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.007477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:36.999318 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-credential-socket\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.007477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.002875 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-certs\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.007477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.007028 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:37.091055 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.090986 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-envoy\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.093491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.092983 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-podinfo\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.126243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.125083 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-token\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.143812 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.143743 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsf6c\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-kube-api-access-nsf6c\") pod \"e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.400155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.400113 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:37.691439 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.691361 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:37.766607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.766577 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf"] Oct 31 00:47:37.768505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.768433 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:37.768617 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.768541 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:37.768617 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:37.768585 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:38.636705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:38.636645 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:38.651043 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:38.651008 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" event={"ID":"f7d3430e-a53d-4202-acd3-bdc2a1faa943","Type":"ContainerStarted","Data":"8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589"} Oct 31 00:47:38.651248 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:38.651068 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" event={"ID":"f7d3430e-a53d-4202-acd3-bdc2a1faa943","Type":"ContainerStarted","Data":"489a35c98d7a80604ad318551707b1add0fe1bfbde2c43f2cd44f469d129a021"} Oct 31 00:47:39.401222 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:39.401183 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:39.407598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:39.407574 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:39.464075 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:39.464006 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podStartSLOduration=3.463988923 podStartE2EDuration="3.463988923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:38.707671482 +0000 UTC m=+1676.405785258" watchObservedRunningTime="2026-10-31 00:47:39.463988923 +0000 UTC m=+1677.162102688" Oct 31 00:47:39.637405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:39.637370 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:39.684187 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:39.684105 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:39.688503 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:39.685795 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:47:40.573599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:40.573565 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:40.739579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:40.734821 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:41.376485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:41.374424 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:41.558188 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:41.558158 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:41.652384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:41.652292 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 00:47:42.078569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:42.078529 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk"] Oct 31 00:47:42.083113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:42.083080 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247318e4-xjwzk"] Oct 31 00:47:42.091390 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:42.091360 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:42.387570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:42.387477 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:42.445306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:42.445271 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ed1645-c162-4303-812c-86a69a125e20" path="/var/lib/kubelet/pods/22ed1645-c162-4303-812c-86a69a125e20/volumes" Oct 31 00:47:44.501601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:44.501557 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:44.653021 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:44.652980 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:44.835485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:44.818647 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.008501 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:44.988411 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.297963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.297790 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf"] Oct 31 00:47:45.313734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.313041 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.341463 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.332504 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf"] Oct 31 00:47:45.353342 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.353259 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.405509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.405474 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f2bdf3-openshift-default-dockercfg-xzqfh\"" Oct 31 00:47:45.427069 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.426716 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.427069 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.426842 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-envoy\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.427347 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.427136 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-certs\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.427347 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.427212 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-token\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.427482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.427358 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-data\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.427482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.427415 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.427594 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.427554 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz9z8\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-kube-api-access-tz9z8\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.427728 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.427711 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-podinfo\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.427816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.427781 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-credential-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.529900 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.529853 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.530402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.529955 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-envoy\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.530402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.530127 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-certs\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.530402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.530184 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-token\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.530402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.530319 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-data\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.530402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.530334 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.530402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.530358 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.530766 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.530666 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-certs\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.530906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.530881 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-data\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.530961 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.530924 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9z8\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-kube-api-access-tz9z8\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.531087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.531044 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-podinfo\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.531145 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.531098 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-credential-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.531235 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.531096 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istiod-ca-cert\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.531601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.531580 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-credential-socket\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.532923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.532898 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-envoy\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.533988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.533972 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-podinfo\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.552229 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.552147 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-token\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.552491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.552430 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9z8\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-kube-api-access-tz9z8\") pod \"e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:45.676671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:45.676624 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:46.013123 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.013090 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf"] Oct 31 00:47:46.014155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.014083 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:46.014279 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.014213 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:46.014279 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.014259 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:46.076858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.076827 5978 scope.go:122] "RemoveContainer" containerID="23ecca57b075ffbba777c914e5d6de9c554760eff08520014c706c71f57c59da" Oct 31 00:47:46.091020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.090994 5978 scope.go:122] "RemoveContainer" containerID="6682e18f55acd51fd8f9c2975bed4acf52a885b6a61d02e9655b2cf910229e85" Oct 31 00:47:46.108979 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.107627 5978 scope.go:122] "RemoveContainer" containerID="168a5a52a8f252d9e1c328fc8789653d5ac9735d6c236ab29218aa8536ae299a" Oct 31 00:47:46.121801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.121776 5978 scope.go:122] "RemoveContainer" containerID="c28cf00cd168056f23226b01c9658d88af87cbf61d7a95afc9cff1b87e32d054" Oct 31 00:47:46.134309 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.134282 5978 scope.go:122] "RemoveContainer" containerID="6425ef992c4154e20ebe349580e75e3b585d2a6f558a57255a2b52e34a51ec20" Oct 31 00:47:46.909573 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.909534 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" event={"ID":"87e36a34-57af-4fa0-8adf-a5b67622a2a5","Type":"ContainerStarted","Data":"9dea4fbdac5f69de3a40ddf7ab813f2fc709796b35dcd7facdeba1170d2e88b7"} Oct 31 00:47:46.910043 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.909590 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" event={"ID":"87e36a34-57af-4fa0-8adf-a5b67622a2a5","Type":"ContainerStarted","Data":"1bf6d5f11c5eb9b3d98caa399bbd05784718207998a042f4d4c49679286328ad"} Oct 31 00:47:46.937089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.936741 5978 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:46.937089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.936809 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:46.966748 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.966694 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" podStartSLOduration=1.966678463 podStartE2EDuration="1.966678463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:46.960913813 +0000 UTC m=+1684.659027579" watchObservedRunningTime="2026-10-31 00:47:46.966678463 +0000 UTC m=+1684.664792228" Oct 31 00:47:46.975822 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:46.975787 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.144770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:47.144735 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.330384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:47.330335 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.677055 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:47.676977 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:47.678891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:47.678852 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:47:47.679086 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:47.678937 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 00:47:48.677997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:48.677956 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:47:48.678431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:48.678023 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 00:47:49.285517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:49.282797 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:49.677406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:49.677365 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:47:49.677726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:49.677435 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 00:47:50.053602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:50.053510 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l"] Oct 31 00:47:50.061494 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:50.061448 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d20d0b-gts9l"] Oct 31 00:47:50.445304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:50.445269 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf96ab3c-a349-4ff5-b535-1653bb0ddda7" path="/var/lib/kubelet/pods/cf96ab3c-a349-4ff5-b535-1653bb0ddda7/volumes" Oct 31 00:47:50.677513 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:50.677472 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:47:50.677716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:50.677534 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 00:47:51.145926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:51.145872 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:51.677961 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:51.677923 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:47:51.678157 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:51.678009 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 00:47:52.321872 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:52.321827 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:52.682233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:52.682151 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:52.682719 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:52.682702 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:52.683584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:52.683559 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 00:47:52.903767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:52.903720 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:54.559094 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:54.559052 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:54.706010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:54.705966 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:55.937600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:55.937557 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:56.026856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:56.026822 5978 generic.go:356] "Generic (PLEG): container finished" podID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerID="1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942" exitCode=0 Oct 31 00:47:56.027043 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:56.026890 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" event={"ID":"9171fc04-6c22-4ca2-a581-bbff5c77205a","Type":"ContainerDied","Data":"1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942"} Oct 31 00:47:56.027442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:56.027423 5978 scope.go:122] "RemoveContainer" containerID="1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942" Oct 31 00:47:56.029371 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:56.029340 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:56.029495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:56.029416 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:56.029495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:56.029447 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:56.291975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:56.291880 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:57.045663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:57.045624 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" event={"ID":"9171fc04-6c22-4ca2-a581-bbff5c77205a","Type":"ContainerStarted","Data":"dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385"} Oct 31 00:47:57.797633 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:57.797599 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:47:57.798430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:57.798399 5978 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:47:57.798570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:57.798498 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:47:57.998271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:57.998239 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:58.186642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:58.186610 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:58.745813 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:58.745769 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:58.813479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:58.797746 5978 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:47:58.813479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:58.797817 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:47:58.923476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:58.923040 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:59.756475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:59.750495 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:59.800477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:59.798869 5978 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:47:59.800477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:47:59.798935 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:48:00.065607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:00.065441 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:00.802439 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:00.801927 5978 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:48:00.802439 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:00.802012 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:48:00.896609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:00.895600 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:48:01.922484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:01.892486 5978 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:48:01.936003 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:01.892560 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:48:02.799477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:02.799329 5978 patch_prober.go:29] interesting pod/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:48:02.799477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:02.799404 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:48:03.106584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:03.106530 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:03.383532 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:48:03.377963 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:48:04.343270 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.340619 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:04.513313 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.513274 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:04.530980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.530946 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:48:04.560821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.560786 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:48:04.963479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.957041 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.963479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.958848 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:05.011535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.981591 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:05.011535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.981716 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 00:48:05.011535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.990164 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 00:48:05.011535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.990731 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:48:05.011535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:04.998550 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 00:48:05.111596 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:05.111553 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:05.837690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:05.837647 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:06.176677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:06.176585 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:06.353741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:06.353704 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:07.653599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:07.653556 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:07.976467 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:07.976359 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:09.465719 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:09.465679 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:09.632919 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:09.632883 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:09.915540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:09.915483 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:10.111122 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:10.111076 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:10.724882 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:10.724847 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:11.823072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:11.823032 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:12.904696 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:12.904661 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.055330 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:13.055296 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt"] Oct 31 00:48:13.064870 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:13.064839 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e5c3c-g9hvt"] Oct 31 00:48:13.445743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:13.445701 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.650877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:13.650835 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.810101 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:13.809984 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:14.039297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:14.039254 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v"] Oct 31 00:48:14.049065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:14.049028 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ce6613-t5c7v"] Oct 31 00:48:14.428893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:14.428859 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9ddac9-2887-4aac-a8f9-85e774801b36" path="/var/lib/kubelet/pods/6e9ddac9-2887-4aac-a8f9-85e774801b36/volumes" Oct 31 00:48:14.429441 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:14.429412 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac" path="/var/lib/kubelet/pods/7cf78a37-4fd9-426d-b6ec-1132ef1ed4ac/volumes" Oct 31 00:48:14.472637 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:14.472596 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:14.667546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:14.667507 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:16.142100 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.141907 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:16.327845 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.327804 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:16.465954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.465872 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf"] Oct 31 00:48:16.469901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.469870 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:16.484549 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.484525 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e166ca\"" Oct 31 00:48:16.493885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.493833 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e166ca-dockercfg-jkgzm\"" Oct 31 00:48:16.500903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.500861 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf"] Oct 31 00:48:16.510295 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.510260 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:16.593480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.583224 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fgv8\" (UniqueName: \"kubernetes.io/projected/26e6fb79-dceb-45cd-becf-978886405174-kube-api-access-2fgv8\") pod \"payload-processing-e2e-api-a-e166ca-7974df756c-ltngf\" (UID: \"26e6fb79-dceb-45cd-becf-978886405174\") " pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:16.593480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.583656 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e6fb79-dceb-45cd-becf-978886405174-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e166ca-7974df756c-ltngf\" (UID: \"26e6fb79-dceb-45cd-becf-978886405174\") " pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:16.740949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.691468 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e6fb79-dceb-45cd-becf-978886405174-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e166ca-7974df756c-ltngf\" (UID: \"26e6fb79-dceb-45cd-becf-978886405174\") " pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:16.740949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.691674 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgv8\" (UniqueName: \"kubernetes.io/projected/26e6fb79-dceb-45cd-becf-978886405174-kube-api-access-2fgv8\") pod \"payload-processing-e2e-api-a-e166ca-7974df756c-ltngf\" (UID: \"26e6fb79-dceb-45cd-becf-978886405174\") " pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:16.740949 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.693064 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e6fb79-dceb-45cd-becf-978886405174-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e166ca-7974df756c-ltngf\" (UID: \"26e6fb79-dceb-45cd-becf-978886405174\") " pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:16.747797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.747698 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgv8\" (UniqueName: \"kubernetes.io/projected/26e6fb79-dceb-45cd-becf-978886405174-kube-api-access-2fgv8\") pod \"payload-processing-e2e-api-a-e166ca-7974df756c-ltngf\" (UID: \"26e6fb79-dceb-45cd-becf-978886405174\") " pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:16.787031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.784383 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:16.822432 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.822402 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2"] Oct 31 00:48:16.847523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.845916 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:16.849507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.848971 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2"] Oct 31 00:48:16.927740 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.927687 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2\" (UID: \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:16.928095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:16.928069 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g949\" (UniqueName: \"kubernetes.io/projected/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-kube-api-access-9g949\") pod \"payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2\" (UID: \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:17.029291 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.029190 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g949\" (UniqueName: \"kubernetes.io/projected/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-kube-api-access-9g949\") pod \"payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2\" (UID: \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:17.029495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.029377 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2\" (UID: \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:17.030625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.030590 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2\" (UID: \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:17.051749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.051713 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g949\" (UniqueName: \"kubernetes.io/projected/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-kube-api-access-9g949\") pod \"payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2\" (UID: \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:17.191180 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.191140 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:17.351034 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.351000 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf"] Oct 31 00:48:17.351311 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:48:17.351287 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e6fb79_dceb_45cd_becf_978886405174.slice/crio-94ac96a57dee3624bb7358da214b7ba0de153753597a0354f1bfb5b296b39912 WatchSource:0}: Error finding container 94ac96a57dee3624bb7358da214b7ba0de153753597a0354f1bfb5b296b39912: Status 404 returned error can't find the container with id 94ac96a57dee3624bb7358da214b7ba0de153753597a0354f1bfb5b296b39912 Oct 31 00:48:17.505890 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.505855 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2"] Oct 31 00:48:17.506705 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:48:17.506659 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3175f116_c972_47f5_bc69_1f3ee3fd2ac7.slice/crio-04af22c303c31d5e14b17e767b03dae0850ba82092e1980bd36ad7fd6d5c0d46 WatchSource:0}: Error finding container 04af22c303c31d5e14b17e767b03dae0850ba82092e1980bd36ad7fd6d5c0d46: Status 404 returned error can't find the container with id 04af22c303c31d5e14b17e767b03dae0850ba82092e1980bd36ad7fd6d5c0d46 Oct 31 00:48:17.727688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.727588 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" event={"ID":"3175f116-c972-47f5-bc69-1f3ee3fd2ac7","Type":"ContainerStarted","Data":"c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922"} Oct 31 00:48:17.727688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.727658 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" event={"ID":"3175f116-c972-47f5-bc69-1f3ee3fd2ac7","Type":"ContainerStarted","Data":"04af22c303c31d5e14b17e767b03dae0850ba82092e1980bd36ad7fd6d5c0d46"} Oct 31 00:48:17.727917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.727704 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:17.729759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.729725 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" event={"ID":"26e6fb79-dceb-45cd-becf-978886405174","Type":"ContainerStarted","Data":"fd029dd31a4051666a0750759f2e83e042b54de2fab006e302d0a04b8d8357f2"} Oct 31 00:48:17.729917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.729787 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" event={"ID":"26e6fb79-dceb-45cd-becf-978886405174","Type":"ContainerStarted","Data":"94ac96a57dee3624bb7358da214b7ba0de153753597a0354f1bfb5b296b39912"} Oct 31 00:48:17.729917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.729829 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:17.774067 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.773996 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" podStartSLOduration=1.773971917 podStartE2EDuration="1.773971917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:17.765060535 +0000 UTC m=+1715.463174303" watchObservedRunningTime="2026-10-31 00:48:17.773971917 +0000 UTC m=+1715.472085683" Oct 31 00:48:17.801837 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.801779 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" podStartSLOduration=1.8017635159999998 podStartE2EDuration="1.801763516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:17.801088293 +0000 UTC m=+1715.499202057" watchObservedRunningTime="2026-10-31 00:48:17.801763516 +0000 UTC m=+1715.499877309" Oct 31 00:48:17.958142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:17.958096 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:18.127029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:18.126992 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:18.416579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:18.414424 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:19.696921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:19.696885 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:19.964024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:19.963938 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:20.831908 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:20.827697 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:22.555045 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:22.554989 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:23.467713 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:23.467670 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:23.627737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:23.627690 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:24.032125 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:24.032083 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm"] Oct 31 00:48:24.040094 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:24.040064 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034b1502-vflpm"] Oct 31 00:48:24.203888 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:24.203851 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:24.429014 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:24.428976 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13dbd9c8-fc21-4b8a-b908-3be71eea3960" path="/var/lib/kubelet/pods/13dbd9c8-fc21-4b8a-b908-3be71eea3960/volumes" Oct 31 00:48:25.786040 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:25.786003 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.379662 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:27.379622 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:28.747169 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:28.747133 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:48:28.747612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:28.747203 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:48:28.984214 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:28.984177 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:30.571841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:30.571804 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:30.731729 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:30.731694 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:30.872933 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:30.872832 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:32.617701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:32.617638 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:33.184081 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:48:33.184043 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:48:34.307862 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:34.307797 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:35.911126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:35.911085 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:37.512487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:37.512379 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:39.140832 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:39.140800 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:41.150715 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.150673 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj"] Oct 31 00:48:41.154950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.154925 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.180529 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.180495 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e166ca-serving-cert\"" Oct 31 00:48:41.185791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.185762 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj"] Oct 31 00:48:41.281285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.281237 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:41.306950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.306909 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-metrics-certs\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.307136 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.306978 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-maas-api-tls\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.307192 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.307160 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7603dd32-be15-4bfa-ae0f-c33908131809-openshift-service-ca\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.307290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.307274 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwwf\" (UniqueName: \"kubernetes.io/projected/7603dd32-be15-4bfa-ae0f-c33908131809-kube-api-access-xhwwf\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.409973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.409874 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-metrics-certs\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.409973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.409954 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-maas-api-tls\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.410243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.410061 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7603dd32-be15-4bfa-ae0f-c33908131809-openshift-service-ca\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.410243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.410132 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwwf\" (UniqueName: \"kubernetes.io/projected/7603dd32-be15-4bfa-ae0f-c33908131809-kube-api-access-xhwwf\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.412289 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.412254 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7603dd32-be15-4bfa-ae0f-c33908131809-openshift-service-ca\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.414264 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.414236 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-maas-api-tls\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.416947 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.416921 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-metrics-certs\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.456812 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.452252 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwwf\" (UniqueName: \"kubernetes.io/projected/7603dd32-be15-4bfa-ae0f-c33908131809-kube-api-access-xhwwf\") pod \"maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.478211 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.478146 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:41.499488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:41.496513 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:42.151556 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:42.151404 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj"] Oct 31 00:48:42.842095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:42.802731 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:43.116804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:43.115265 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" event={"ID":"7603dd32-be15-4bfa-ae0f-c33908131809","Type":"ContainerStarted","Data":"08f5b574f3f742915f0f44d2858df901e1f3873b611fed16b7848f384e4b1907"} Oct 31 00:48:43.116804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:43.116649 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:43.174020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:43.173953 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" podStartSLOduration=1.826052215 podStartE2EDuration="2.173926796s" totalImagesPullingTime="347.874581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:43.168335258 +0000 UTC m=+1740.866449023" watchObservedRunningTime="2026-10-31 00:48:43.173926796 +0000 UTC m=+1740.872040562" Oct 31 00:48:43.317985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:43.317945 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:43.893390 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:43.893186 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:44.145833 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:44.145559 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:44.217790 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:44.217747 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" event={"ID":"7603dd32-be15-4bfa-ae0f-c33908131809","Type":"ContainerStarted","Data":"150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca"} Oct 31 00:48:44.350484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:44.349929 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:44.971705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:44.970639 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:44.972280 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:44.972248 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:44.983286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:44.982543 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:45.028478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:44.993667 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:45.053614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:45.053565 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 00:48:45.142493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:45.091770 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded" Oct 31 00:48:45.211477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:45.191101 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g"] Oct 31 00:48:45.363532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:45.320137 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889dfd10-mph6g"] Oct 31 00:48:45.409544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:45.409477 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:45.614406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:45.605177 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:45.719292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:45.719231 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:45.927735 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:45.927599 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:45.957874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:45.957788 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:46.000644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:46.000594 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:46.001636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:46.001609 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:48:46.405509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:46.402781 5978 scope.go:122] "RemoveContainer" containerID="cde965b70a05192bcd53bd7a22b83256714a4be51ae0281226ef46e42aea96b9" Oct 31 00:48:46.409836 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:46.409777 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 31 00:48:46.598805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:46.598767 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:46.607490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:46.605905 5978 scope.go:122] "RemoveContainer" containerID="0ffbd771fe134aade95b22075d553a7afcdce5e57a5081699fa583d54c802b8c" Oct 31 00:48:46.676578 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:46.676542 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c006ff-516f-445c-8c98-62927442b391" path="/var/lib/kubelet/pods/c7c006ff-516f-445c-8c98-62927442b391/volumes" Oct 31 00:48:46.677424 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:46.677404 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 00:48:46.688076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:46.688027 5978 scope.go:122] "RemoveContainer" containerID="34ded859130c2ed7d3138700af1cadc835141d3b9ec299b33ab1705d8067d38d" Oct 31 00:48:47.062929 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:47.062895 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2"] Oct 31 00:48:47.067701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:47.067667 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c14659-qb8k2"] Oct 31 00:48:48.338413 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.338375 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75"] Oct 31 00:48:48.344704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.344677 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.351670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.351641 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75"] Oct 31 00:48:48.358153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.358126 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fa2c15c1-serving-cert\"" Oct 31 00:48:48.447344 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.447313 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0d28cf-4936-4a92-b06b-b2f2f6499229" path="/var/lib/kubelet/pods/9a0d28cf-4936-4a92-b06b-b2f2f6499229/volumes" Oct 31 00:48:48.447992 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.447971 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-openshift-service-ca\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.448054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.448022 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv99p\" (UniqueName: \"kubernetes.io/projected/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-kube-api-access-hv99p\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.448150 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.448137 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-maas-api-tls\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.448186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.448167 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-metrics-certs\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.454745 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.454711 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:48.549888 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.549856 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-openshift-service-ca\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.550059 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.549939 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv99p\" (UniqueName: \"kubernetes.io/projected/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-kube-api-access-hv99p\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.550124 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.550067 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-maas-api-tls\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.550181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.550120 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-metrics-certs\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.550782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.550758 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-openshift-service-ca\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.552765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.552743 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-metrics-certs\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.552874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.552793 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-maas-api-tls\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.593543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.580425 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv99p\" (UniqueName: \"kubernetes.io/projected/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-kube-api-access-hv99p\") pod \"maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:48.610751 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.610562 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:48.666114 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:48.665803 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:49.032120 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:49.032087 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75"] Oct 31 00:48:49.700203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:49.700102 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" event={"ID":"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02","Type":"ContainerStarted","Data":"eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e"} Oct 31 00:48:49.700680 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:49.700569 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" event={"ID":"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02","Type":"ContainerStarted","Data":"8619a3571f077703cdbae6509131c6f6665844414f710bbf49d9d1a4590f8b88"} Oct 31 00:48:49.700680 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:49.700598 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:49.744824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:49.744768 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" podStartSLOduration=1.410792323 podStartE2EDuration="1.744750636s" totalImagesPullingTime="333.958313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:49.737801766 +0000 UTC m=+1747.435915532" watchObservedRunningTime="2026-10-31 00:48:49.744750636 +0000 UTC m=+1747.442864822" Oct 31 00:48:50.431591 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:50.431560 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:48:50.588010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:50.587944 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:51.032399 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:51.032357 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:51.209781 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:51.209741 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:52.725491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:52.725411 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:52.899852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:52.899807 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:53.109758 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.109646 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x"] Oct 31 00:48:53.296785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.296747 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x"] Oct 31 00:48:53.296996 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.296884 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:48:53.315660 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.315621 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770d1ffc-662a-4979-a708-ba009efd843e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x\" (UID: \"770d1ffc-662a-4979-a708-ba009efd843e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:48:53.315992 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.315965 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzwx\" (UniqueName: \"kubernetes.io/projected/770d1ffc-662a-4979-a708-ba009efd843e-kube-api-access-nvzwx\") pod \"payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x\" (UID: \"770d1ffc-662a-4979-a708-ba009efd843e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:48:53.320134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.320111 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-714f8f\"" Oct 31 00:48:53.320258 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.320160 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-714f8f-dockercfg-9zmdg\"" Oct 31 00:48:53.417448 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.417349 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzwx\" (UniqueName: \"kubernetes.io/projected/770d1ffc-662a-4979-a708-ba009efd843e-kube-api-access-nvzwx\") pod \"payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x\" (UID: \"770d1ffc-662a-4979-a708-ba009efd843e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:48:53.417685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.417640 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770d1ffc-662a-4979-a708-ba009efd843e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x\" (UID: \"770d1ffc-662a-4979-a708-ba009efd843e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:48:53.423005 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.422975 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770d1ffc-662a-4979-a708-ba009efd843e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x\" (UID: \"770d1ffc-662a-4979-a708-ba009efd843e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:48:53.462821 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.462709 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzwx\" (UniqueName: \"kubernetes.io/projected/770d1ffc-662a-4979-a708-ba009efd843e-kube-api-access-nvzwx\") pod \"payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x\" (UID: \"770d1ffc-662a-4979-a708-ba009efd843e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:48:53.643292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:53.643250 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:48:54.705108 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:54.705061 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x"] Oct 31 00:48:54.718683 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:48:54.718636 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770d1ffc_662a_4979_a708_ba009efd843e.slice/crio-4fb281c30609dfe62d843dd6cd07ddd703f019c721b2daba86861572ca9034ff WatchSource:0}: Error finding container 4fb281c30609dfe62d843dd6cd07ddd703f019c721b2daba86861572ca9034ff: Status 404 returned error can't find the container with id 4fb281c30609dfe62d843dd6cd07ddd703f019c721b2daba86861572ca9034ff Oct 31 00:48:54.764285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:54.764251 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" event={"ID":"770d1ffc-662a-4979-a708-ba009efd843e","Type":"ContainerStarted","Data":"4fb281c30609dfe62d843dd6cd07ddd703f019c721b2daba86861572ca9034ff"} Oct 31 00:48:55.076019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:54.907938 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:55.210675 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.210639 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": read tcp 10.134.0.2:53180->10.134.0.137:15021: read: connection reset by peer" start-of-body= Oct 31 00:48:55.210852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.210709 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": read tcp 10.134.0.2:53180->10.134.0.137:15021: read: connection reset by peer" Oct 31 00:48:55.727429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.727390 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:48:55.776329 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.776295 5978 generic.go:356] "Generic (PLEG): container finished" podID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerID="8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589" exitCode=0 Oct 31 00:48:55.776530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.776369 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" event={"ID":"f7d3430e-a53d-4202-acd3-bdc2a1faa943","Type":"ContainerDied","Data":"8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589"} Oct 31 00:48:55.777078 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.777054 5978 scope.go:122] "RemoveContainer" containerID="8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589" Oct 31 00:48:55.779273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.779248 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" event={"ID":"770d1ffc-662a-4979-a708-ba009efd843e","Type":"ContainerStarted","Data":"991f5d42b558bdf620178c373a80dc7643db6f04e2ea3e73f0e636e4377026d5"} Oct 31 00:48:55.779386 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.779325 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:48:55.779523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.779498 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:55.779591 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.779565 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:55.779646 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.779607 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:55.862630 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:55.862556 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" podStartSLOduration=2.862533984 podStartE2EDuration="2.862533984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:55.85490754 +0000 UTC m=+1753.553021305" watchObservedRunningTime="2026-10-31 00:48:55.862533984 +0000 UTC m=+1753.560647750" Oct 31 00:48:56.571968 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:56.571930 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:56.790888 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:56.790847 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" event={"ID":"f7d3430e-a53d-4202-acd3-bdc2a1faa943","Type":"ContainerStarted","Data":"f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40"} Oct 31 00:48:57.402071 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:57.402038 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:48:57.402293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:57.402236 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:48:57.402357 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:57.402293 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:48:58.203840 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:58.203801 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:58.401353 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:58.401319 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:48:58.401579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:58.401396 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:48:59.401332 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:59.401281 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:48:59.401929 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:59.401398 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:48:59.572621 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:59.572581 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:59.890814 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:48:59.890780 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:00.401220 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:00.401185 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:49:00.401444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:00.401249 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:49:01.401336 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:01.401296 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:49:01.401549 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:01.401348 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:49:01.456976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:01.456941 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:02.400631 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:02.400590 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:49:02.400828 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:02.400656 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:49:03.113476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:03.113257 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:03.217540 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:49:03.217447 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:49:03.405884 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:03.405808 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:49:03.406376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:03.406351 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:49:03.406986 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:03.406961 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 00:49:04.033895 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:04.033863 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l"] Oct 31 00:49:04.066125 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:04.066087 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f689cbc1-l8r2l"] Oct 31 00:49:04.429879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:04.429842 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e01341-02df-4bf5-8935-4309dde93168" path="/var/lib/kubelet/pods/67e01341-02df-4bf5-8935-4309dde93168/volumes" Oct 31 00:49:04.951103 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:04.951066 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:06.810743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:06.809318 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 00:49:06.819592 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:06.816151 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:07.399799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:07.399758 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:07.589958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:07.589917 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:09.631271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:09.629516 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:10.084287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:10.084244 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:10.797184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:10.797145 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.291875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.291834 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms"] Oct 31 00:49:11.295939 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.295918 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.331098 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.331058 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3b1945-openshift-default-dockercfg-fkhzm\"" Oct 31 00:49:11.449859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.449065 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms"] Oct 31 00:49:11.460074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.459870 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.545553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.544005 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a043e49-6ec4-47b9-8323-d49ee320d95c-istiod-ca-cert\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.545553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.544092 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-envoy\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.545553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.544172 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-data\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.545553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.544212 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vgl\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-kube-api-access-d5vgl\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.545553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.544319 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-certs\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.545553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.544427 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-socket\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.545553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.544531 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-podinfo\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.545553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.544587 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-token\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.545553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.544661 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-credential-socket\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.612475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.612074 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.647062 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.647022 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a043e49-6ec4-47b9-8323-d49ee320d95c-istiod-ca-cert\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.647249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.647082 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-envoy\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.647249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.647149 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-data\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.647249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.647180 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vgl\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-kube-api-access-d5vgl\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.647362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.647268 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-certs\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.647362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.647355 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-socket\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.647423 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.647402 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-podinfo\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.647534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.647470 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-token\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.647534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.647526 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-credential-socket\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.649730 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.649699 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-data\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.649913 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.649786 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a043e49-6ec4-47b9-8323-d49ee320d95c-istiod-ca-cert\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.649913 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.649863 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-credential-socket\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.650579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.650441 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-socket\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.650579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.650535 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-certs\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.651140 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.651121 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-envoy\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.651282 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.651263 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-podinfo\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.701850 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.701817 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-token\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.702064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.702038 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vgl\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-kube-api-access-d5vgl\") pod \"e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:11.912218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:11.912178 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:12.268593 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:12.268559 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms"] Oct 31 00:49:12.268946 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:49:12.268918 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a043e49_6ec4_47b9_8323_d49ee320d95c.slice/crio-47933318921cea5a67e541c9e65fc08f6f091be983f4f1b759d09272af912575 WatchSource:0}: Error finding container 47933318921cea5a67e541c9e65fc08f6f091be983f4f1b759d09272af912575: Status 404 returned error can't find the container with id 47933318921cea5a67e541c9e65fc08f6f091be983f4f1b759d09272af912575 Oct 31 00:49:12.274849 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:12.274799 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:12.275019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:12.274909 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:12.275019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:12.274954 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:13.092825 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:13.092785 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" event={"ID":"3a043e49-6ec4-47b9-8323-d49ee320d95c","Type":"ContainerStarted","Data":"433a159cdd9a7ab4e0cb271cc4f02c1e063610ec787d9d564137350efff32599"} Oct 31 00:49:13.093308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:13.092854 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" event={"ID":"3a043e49-6ec4-47b9-8323-d49ee320d95c","Type":"ContainerStarted","Data":"47933318921cea5a67e541c9e65fc08f6f091be983f4f1b759d09272af912575"} Oct 31 00:49:13.146735 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:13.146670 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" podStartSLOduration=2.146613114 podStartE2EDuration="2.146613114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:13.143670917 +0000 UTC m=+1770.841784693" watchObservedRunningTime="2026-10-31 00:49:13.146613114 +0000 UTC m=+1770.844726880" Oct 31 00:49:13.320600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:13.320567 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:13.922421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:13.922383 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:13.929436 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:13.929397 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:14.102801 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:14.102772 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:14.103935 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:14.103909 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:49:15.248044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:15.240859 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:15.685891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:15.685848 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:15.883248 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:15.883212 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:16.039290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:16.039204 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp"] Oct 31 00:49:16.047330 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:16.047300 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45f4a89b-2hvqp"] Oct 31 00:49:16.429767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:16.429735 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f" path="/var/lib/kubelet/pods/c0b2e26b-70ad-49f2-a3b8-94df8d5aeb4f/volumes" Oct 31 00:49:17.036681 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:17.036645 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:18.256202 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:18.256152 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:18.593707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:18.593672 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:19.359671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:19.359634 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:19.540151 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:19.540108 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:19.705576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:19.705480 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:20.825914 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:20.825878 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:21.590303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:21.590267 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:21.744269 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:21.744233 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:22.110869 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:22.110828 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:22.913607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:22.913567 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:23.752080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.752049 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr"] Oct 31 00:49:23.756843 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.756822 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.770148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.770125 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-714f8f-serving-cert\"" Oct 31 00:49:23.787685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.787652 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr"] Oct 31 00:49:23.821997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.821959 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8t4x\" (UniqueName: \"kubernetes.io/projected/4cd2a420-72b7-4763-be0f-888be3754b5b-kube-api-access-b8t4x\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.822188 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.822135 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.822239 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.822187 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.822276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.822238 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a420-72b7-4763-be0f-888be3754b5b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.918978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.918936 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:23.923881 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.923843 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8t4x\" (UniqueName: \"kubernetes.io/projected/4cd2a420-72b7-4763-be0f-888be3754b5b-kube-api-access-b8t4x\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.924041 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.923942 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.924041 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.923966 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.924041 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.923994 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a420-72b7-4763-be0f-888be3754b5b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.924177 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:49:23.924138 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-serving-cert: secret "maas-api-e2e-shared-b-w6-714f8f-serving-cert" not found Oct 31 00:49:23.925769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.925747 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a420-72b7-4763-be0f-888be3754b5b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.926681 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.926650 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:23.929270 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:49:23.929246 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls podName:4cd2a420-72b7-4763-be0f-888be3754b5b nodeName:}" failed. No retries permitted until 2026-10-31 00:49:24.426617799 +0000 UTC m=+1782.124731542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls") pod "maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" (UID: "4cd2a420-72b7-4763-be0f-888be3754b5b") : secret "maas-api-e2e-shared-b-w6-714f8f-serving-cert" not found Oct 31 00:49:23.945295 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:23.945263 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8t4x\" (UniqueName: \"kubernetes.io/projected/4cd2a420-72b7-4763-be0f-888be3754b5b-kube-api-access-b8t4x\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:24.087896 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:24.087859 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:24.435765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:24.435672 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:24.438176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:24.438154 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:24.590300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:24.590263 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:24.673738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:24.673701 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:24.988791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:24.988762 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr"] Oct 31 00:49:24.990067 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:49:24.990032 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd2a420_72b7_4763_be0f_888be3754b5b.slice/crio-bf19765cc9ef7319b5d6c8508ec3f0fba720e10e09ef5d84fb8f79015545515c WatchSource:0}: Error finding container bf19765cc9ef7319b5d6c8508ec3f0fba720e10e09ef5d84fb8f79015545515c: Status 404 returned error can't find the container with id bf19765cc9ef7319b5d6c8508ec3f0fba720e10e09ef5d84fb8f79015545515c Oct 31 00:49:25.322445 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:25.322408 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" event={"ID":"4cd2a420-72b7-4763-be0f-888be3754b5b","Type":"ContainerStarted","Data":"bf19765cc9ef7319b5d6c8508ec3f0fba720e10e09ef5d84fb8f79015545515c"} Oct 31 00:49:25.602120 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:25.602078 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:25.758744 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:25.758705 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:26.333881 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:26.333841 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" event={"ID":"4cd2a420-72b7-4763-be0f-888be3754b5b","Type":"ContainerStarted","Data":"c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce"} Oct 31 00:49:26.334410 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:26.334069 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:26.351315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:26.351279 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:26.388997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:26.388185 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" podStartSLOduration=3.089502166 podStartE2EDuration="3.388167476s" totalImagesPullingTime="298.66531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:26.386220642 +0000 UTC m=+1784.084334407" watchObservedRunningTime="2026-10-31 00:49:26.388167476 +0000 UTC m=+1784.086281240" Oct 31 00:49:26.492942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:26.492908 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:28.430180 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:28.430141 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:29.399770 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:29.399726 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:29.578207 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:29.578169 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:30.054439 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:30.054404 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd"] Oct 31 00:49:30.063473 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:30.063427 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63505f52-5czqd"] Oct 31 00:49:30.428891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:30.428854 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8727fc-6db1-40c3-903a-0c61b1aaa810" path="/var/lib/kubelet/pods/9c8727fc-6db1-40c3-903a-0c61b1aaa810/volumes" Oct 31 00:49:30.450267 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:30.450232 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:31.452663 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:31.452606 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:31.647510 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:31.647434 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:32.096535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:32.096487 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg"] Oct 31 00:49:32.103301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:32.103250 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54"] Oct 31 00:49:32.238131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:32.238074 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:32.349133 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:32.349051 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 00:49:32.460771 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:32.460670 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:33.122481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:33.120853 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:33.223786 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:49:33.223741 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:49:33.847002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:33.846964 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:34.935194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:34.935157 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:36.595201 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:36.595169 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:37.104134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:37.104074 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" podUID="be61c2cf-6d4a-40ab-a95b-e4413a1394a7" containerName="payload-pre-processing" containerID="cri-o://3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035" gracePeriod=25 Oct 31 00:49:37.105650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:37.105611 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" podUID="740368e2-d7f8-41ec-ac01-3e5b9b2fa707" containerName="payload-processing" containerID="cri-o://91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e" gracePeriod=25 Oct 31 00:49:38.231881 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:38.231842 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:39.885465 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:39.885410 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:40.025009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:40.024973 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:40.827612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:40.827574 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:41.502864 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:41.502821 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:41.973897 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:41.973852 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" podUID="be61c2cf-6d4a-40ab-a95b-e4413a1394a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:49:41.974089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:41.973921 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" podUID="740368e2-d7f8-41ec-ac01-3e5b9b2fa707" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:49:43.199218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:43.199178 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:44.792164 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:44.792129 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:45.716140 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:45.716100 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:46.591275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:46.591238 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:46.740754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:46.740707 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:47.009047 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:47.008962 5978 scope.go:122] "RemoveContainer" containerID="1135d2ea864d6da16dce6245098b6cc6d190e8f6119af94f082ba5ee6b3bcc97" Oct 31 00:49:47.055651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:47.055591 5978 scope.go:122] "RemoveContainer" containerID="3358b8673ccaf34001141f6eb41b504a8d1cc55906c7de094dacacb39423cebd" Oct 31 00:49:47.070089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:47.070040 5978 scope.go:122] "RemoveContainer" containerID="cf64dfecad3b44f0b0499af112cf703175d52b9878cbf04cbbb5e99be0b431b8" Oct 31 00:49:47.083197 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:47.083174 5978 scope.go:122] "RemoveContainer" containerID="bb021fd8d000b3d00f0ab92bb7a76e28913a8f152c99856467faffdcf0b542a0" Oct 31 00:49:47.101120 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:47.101095 5978 scope.go:122] "RemoveContainer" containerID="5def39e58a2e202ac6b2c128610a7a667f30536a2ccdc4f315d0c8498ce032cd" Oct 31 00:49:47.114871 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:47.114835 5978 scope.go:122] "RemoveContainer" containerID="1629274eb20631ba751f9aca545f31a3f476e04e88de626de39258f1ab1c4352" Oct 31 00:49:47.460338 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:47.460299 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.188859 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:48.188821 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.335339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:48.335298 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.564759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:48.564668 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:49.965577 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:49.965530 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.234736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:50.234645 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:49:50.238444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:50.238417 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:49:51.033749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:51.033719 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g"] Oct 31 00:49:51.043066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:51.043033 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86619be-25f5g"] Oct 31 00:49:51.663398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:51.663349 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:51.973780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:51.973678 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" podUID="be61c2cf-6d4a-40ab-a95b-e4413a1394a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:49:51.973780 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:51.973723 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" podUID="740368e2-d7f8-41ec-ac01-3e5b9b2fa707" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:49:52.428999 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:52.428960 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad3a0c8-1c55-408e-9523-7ea822456671" path="/var/lib/kubelet/pods/fad3a0c8-1c55-408e-9523-7ea822456671/volumes" Oct 31 00:49:53.528518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:53.528480 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:55.696912 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:55.696870 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:57.453968 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.453935 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:49:57.464366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.464331 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:49:57.553495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.553439 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-plugins-config-volume\") pod \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\" (UID: \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\") " Oct 31 00:49:57.553681 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.553582 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-kube-api-access-rcmj5\" (UniqueName: \"kubernetes.io/projected/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-kube-api-access-rcmj5\") pod \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\" (UID: \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\") " Oct 31 00:49:57.553681 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.553636 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-kube-api-access-45tdr\" (UniqueName: \"kubernetes.io/projected/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-kube-api-access-45tdr\") pod \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\" (UID: \"740368e2-d7f8-41ec-ac01-3e5b9b2fa707\") " Oct 31 00:49:57.553905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.553889 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-plugins-config-volume\") pod \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\" (UID: \"be61c2cf-6d4a-40ab-a95b-e4413a1394a7\") " Oct 31 00:49:57.553990 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.553907 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-plugins-config-volume" pod "740368e2-d7f8-41ec-ac01-3e5b9b2fa707" (UID: "740368e2-d7f8-41ec-ac01-3e5b9b2fa707"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:57.554219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.554197 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-plugins-config-volume" pod "be61c2cf-6d4a-40ab-a95b-e4413a1394a7" (UID: "be61c2cf-6d4a-40ab-a95b-e4413a1394a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:57.555044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.555016 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:49:57.555044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.555045 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:49:57.558856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.558831 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-kube-api-access-rcmj5" pod "be61c2cf-6d4a-40ab-a95b-e4413a1394a7" (UID: "be61c2cf-6d4a-40ab-a95b-e4413a1394a7"). InnerVolumeSpecName "kube-api-access-rcmj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:57.558982 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.558869 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-kube-api-access-45tdr" pod "740368e2-d7f8-41ec-ac01-3e5b9b2fa707" (UID: "740368e2-d7f8-41ec-ac01-3e5b9b2fa707"). InnerVolumeSpecName "kube-api-access-45tdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:57.644518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.644478 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:57.654015 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.653905 5978 generic.go:356] "Generic (PLEG): container finished" podID="be61c2cf-6d4a-40ab-a95b-e4413a1394a7" containerID="3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035" exitCode=0 Oct 31 00:49:57.654015 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.653963 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" Oct 31 00:49:57.654015 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.653998 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" event={"ID":"be61c2cf-6d4a-40ab-a95b-e4413a1394a7","Type":"ContainerDied","Data":"3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035"} Oct 31 00:49:57.654266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.654058 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54" event={"ID":"be61c2cf-6d4a-40ab-a95b-e4413a1394a7","Type":"ContainerDied","Data":"3b420ea494a91eb741c8f6a9c4b1d178cb20255a4be0868d1bc55802689368cb"} Oct 31 00:49:57.654266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.654098 5978 scope.go:122] "RemoveContainer" containerID="3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035" Oct 31 00:49:57.659905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.656523 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcmj5\" (UniqueName: \"kubernetes.io/projected/be61c2cf-6d4a-40ab-a95b-e4413a1394a7-kube-api-access-rcmj5\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:49:57.659905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.656549 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45tdr\" (UniqueName: \"kubernetes.io/projected/740368e2-d7f8-41ec-ac01-3e5b9b2fa707-kube-api-access-45tdr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:49:57.667428 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.667392 5978 generic.go:356] "Generic (PLEG): container finished" podID="740368e2-d7f8-41ec-ac01-3e5b9b2fa707" containerID="91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e" exitCode=0 Oct 31 00:49:57.667687 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.667440 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" event={"ID":"740368e2-d7f8-41ec-ac01-3e5b9b2fa707","Type":"ContainerDied","Data":"91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e"} Oct 31 00:49:57.667687 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.667446 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" Oct 31 00:49:57.667687 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.667501 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg" event={"ID":"740368e2-d7f8-41ec-ac01-3e5b9b2fa707","Type":"ContainerDied","Data":"36ade3d836ad385aae0e3cb9bd4ce1f3442eec9a8df142831c0bb06ad122664f"} Oct 31 00:49:57.678961 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.678936 5978 scope.go:122] "RemoveContainer" containerID="3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035" Oct 31 00:49:57.679352 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:49:57.679325 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035\": container with ID starting with 3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035 not found: ID does not exist" containerID="3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035" Oct 31 00:49:57.679446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.679360 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035"} err="failed to get container status \"3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035\": rpc error: code = NotFound desc = could not find container \"3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035\": container with ID starting with 3b49d212decca7a7459385f9a316d450b3bf8b585a8c8f00662798474ddb2035 not found: ID does not exist" Oct 31 00:49:57.679446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.679385 5978 scope.go:122] "RemoveContainer" containerID="91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e" Oct 31 00:49:57.697390 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.697361 5978 scope.go:122] "RemoveContainer" containerID="91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e" Oct 31 00:49:57.697865 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:49:57.697842 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e\": container with ID starting with 91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e not found: ID does not exist" containerID="91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e" Oct 31 00:49:57.697973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.697878 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e"} err="failed to get container status \"91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e\": rpc error: code = NotFound desc = could not find container \"91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e\": container with ID starting with 91b155b972f55746f67bdbc81c81c539f6ce9d4c7c9b7cd7b50ec406370ed72e not found: ID does not exist" Oct 31 00:49:57.739171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.738408 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg"] Oct 31 00:49:57.747569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.745831 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-208e9f32-7f6679cdf7-vpmkg"] Oct 31 00:49:57.771414 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.771380 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54"] Oct 31 00:49:57.781903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:57.781865 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-208e9f32-5b998b4dcf-hcc54"] Oct 31 00:49:58.428953 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:58.428871 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740368e2-d7f8-41ec-ac01-3e5b9b2fa707" path="/var/lib/kubelet/pods/740368e2-d7f8-41ec-ac01-3e5b9b2fa707/volumes" Oct 31 00:49:58.429512 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:58.429446 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be61c2cf-6d4a-40ab-a95b-e4413a1394a7" path="/var/lib/kubelet/pods/be61c2cf-6d4a-40ab-a95b-e4413a1394a7/volumes" Oct 31 00:49:59.737972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:49:59.737929 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:01.479317 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:01.479266 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:01.688736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:01.687468 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:01.831708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:01.831678 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:02.736695 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:02.736651 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:03.182757 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:50:03.182720 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:50:03.483318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:03.483195 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:04.080021 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:04.079985 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:04.182436 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:04.182400 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s"] Oct 31 00:50:04.182712 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:04.182680 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" containerID="cri-o://0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765" gracePeriod=30 Oct 31 00:50:04.899636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:04.899515 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:05.289371 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:05.289282 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:05.633095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:05.633047 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:07.428154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:07.428123 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:08.807742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:08.807698 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:08.976370 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:08.976325 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.394609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.394513 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.575656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.575630 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:50:09.663351 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.663307 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7474b67-7515-46f1-9b80-09c2353018a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7474b67-7515-46f1-9b80-09c2353018a0-istiod-ca-cert\") pod \"d7474b67-7515-46f1-9b80-09c2353018a0\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " Oct 31 00:50:09.663561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.663410 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-socket\") pod \"d7474b67-7515-46f1-9b80-09c2353018a0\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " Oct 31 00:50:09.663561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.663465 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-envoy\") pod \"d7474b67-7515-46f1-9b80-09c2353018a0\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " Oct 31 00:50:09.663561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.663505 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-data\") pod \"d7474b67-7515-46f1-9b80-09c2353018a0\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " Oct 31 00:50:09.663765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.663561 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-credential-socket\") pod \"d7474b67-7515-46f1-9b80-09c2353018a0\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " Oct 31 00:50:09.663765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.663606 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7474b67-7515-46f1-9b80-09c2353018a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7474b67-7515-46f1-9b80-09c2353018a0-istio-podinfo\") pod \"d7474b67-7515-46f1-9b80-09c2353018a0\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " Oct 31 00:50:09.663875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.663799 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-certs\") pod \"d7474b67-7515-46f1-9b80-09c2353018a0\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " Oct 31 00:50:09.663875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.663843 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-kube-api-access-gc82d\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-kube-api-access-gc82d\") pod \"d7474b67-7515-46f1-9b80-09c2353018a0\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " Oct 31 00:50:09.663987 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.663904 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-istio-token\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-istio-token\") pod \"d7474b67-7515-46f1-9b80-09c2353018a0\" (UID: \"d7474b67-7515-46f1-9b80-09c2353018a0\") " Oct 31 00:50:09.664393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.664366 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-socket" pod "d7474b67-7515-46f1-9b80-09c2353018a0" (UID: "d7474b67-7515-46f1-9b80-09c2353018a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:09.664686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.664664 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:09.665099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.664853 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-data" pod "d7474b67-7515-46f1-9b80-09c2353018a0" (UID: "d7474b67-7515-46f1-9b80-09c2353018a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:09.667389 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.667330 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-credential-socket" pod "d7474b67-7515-46f1-9b80-09c2353018a0" (UID: "d7474b67-7515-46f1-9b80-09c2353018a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:09.667543 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.667392 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-envoy" pod "d7474b67-7515-46f1-9b80-09c2353018a0" (UID: "d7474b67-7515-46f1-9b80-09c2353018a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:09.667616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.667567 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-certs" pod "d7474b67-7515-46f1-9b80-09c2353018a0" (UID: "d7474b67-7515-46f1-9b80-09c2353018a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:09.667827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.667807 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7474b67-7515-46f1-9b80-09c2353018a0-istiod-ca-cert" pod "d7474b67-7515-46f1-9b80-09c2353018a0" (UID: "d7474b67-7515-46f1-9b80-09c2353018a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:09.668043 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.668018 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-kube-api-access-gc82d" pod "d7474b67-7515-46f1-9b80-09c2353018a0" (UID: "d7474b67-7515-46f1-9b80-09c2353018a0"). InnerVolumeSpecName "kube-api-access-gc82d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:09.668977 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.668953 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7474b67-7515-46f1-9b80-09c2353018a0-istio-podinfo" pod "d7474b67-7515-46f1-9b80-09c2353018a0" (UID: "d7474b67-7515-46f1-9b80-09c2353018a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:09.669936 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.669909 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-istio-token" pod "d7474b67-7515-46f1-9b80-09c2353018a0" (UID: "d7474b67-7515-46f1-9b80-09c2353018a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:09.767808 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.767769 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:09.767808 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.767808 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:09.768130 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.767823 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:09.768130 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.767836 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7474b67-7515-46f1-9b80-09c2353018a0-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:09.768130 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.767848 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7474b67-7515-46f1-9b80-09c2353018a0-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:09.768130 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.767862 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc82d\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-kube-api-access-gc82d\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:09.768130 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.767877 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7474b67-7515-46f1-9b80-09c2353018a0-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:09.768130 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.767887 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7474b67-7515-46f1-9b80-09c2353018a0-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:09.860318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.860280 5978 generic.go:356] "Generic (PLEG): container finished" podID="d7474b67-7515-46f1-9b80-09c2353018a0" containerID="0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765" exitCode=0 Oct 31 00:50:09.860807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.860341 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" Oct 31 00:50:09.860807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.860347 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" event={"ID":"d7474b67-7515-46f1-9b80-09c2353018a0","Type":"ContainerDied","Data":"0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765"} Oct 31 00:50:09.860807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.860499 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s" event={"ID":"d7474b67-7515-46f1-9b80-09c2353018a0","Type":"ContainerDied","Data":"7d5782cdcb635556105d90b13ca54ccbc285911e52e7d623dd85901278152bbb"} Oct 31 00:50:09.860807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.860523 5978 scope.go:122] "RemoveContainer" containerID="0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765" Oct 31 00:50:09.875702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.875678 5978 scope.go:122] "RemoveContainer" containerID="0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765" Oct 31 00:50:09.876053 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:50:09.876032 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765\": container with ID starting with 0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765 not found: ID does not exist" containerID="0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765" Oct 31 00:50:09.876148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.876060 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765"} err="failed to get container status \"0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765\": rpc error: code = NotFound desc = could not find container \"0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765\": container with ID starting with 0f33a5dc7adf75d212aadbd31e784cda9db06720f63ef0700b28283a0cdcc765 not found: ID does not exist" Oct 31 00:50:09.912907 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.912875 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s"] Oct 31 00:50:09.922332 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:09.922303 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-xw55s"] Oct 31 00:50:10.429773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:10.429739 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" path="/var/lib/kubelet/pods/d7474b67-7515-46f1-9b80-09c2353018a0/volumes" Oct 31 00:50:10.559325 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:10.559284 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:10.716760 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:10.716665 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:11.800368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:11.800328 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:13.810536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:13.810495 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:15.816930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:15.816894 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:17.345447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:17.345403 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:17.908851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:17.908758 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:18.230246 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:18.230150 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:18.381972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:18.381922 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:19.230958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.230924 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk"] Oct 31 00:50:19.231263 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.231159 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" containerID="cri-o://ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c" gracePeriod=30 Oct 31 00:50:19.569640 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.569564 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85"] Oct 31 00:50:19.571216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.570684 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" containerID="cri-o://4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7" gracePeriod=30 Oct 31 00:50:19.575651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.575626 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb"] Oct 31 00:50:19.575839 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.575819 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" containerID="cri-o://a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0" gracePeriod=30 Oct 31 00:50:19.581079 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.581045 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp"] Oct 31 00:50:19.581247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.581225 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" containerID="cri-o://96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb" gracePeriod=30 Oct 31 00:50:19.586580 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.586551 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l"] Oct 31 00:50:19.586790 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:19.586763 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" containerID="cri-o://3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11" gracePeriod=30 Oct 31 00:50:20.241490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:20.241422 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:21.907945 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:21.907873 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:22.753778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:22.753733 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:23.712009 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:23.711897 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:24.559396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.559363 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:50:24.578723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.578688 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/827f550b-a8d9-4074-b34d-551ae660d799-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827f550b-a8d9-4074-b34d-551ae660d799-istio-podinfo\") pod \"827f550b-a8d9-4074-b34d-551ae660d799\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " Oct 31 00:50:24.578930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.578889 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-certs\") pod \"827f550b-a8d9-4074-b34d-551ae660d799\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " Oct 31 00:50:24.579008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.578982 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-kube-api-access-46zjr\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-kube-api-access-46zjr\") pod \"827f550b-a8d9-4074-b34d-551ae660d799\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " Oct 31 00:50:24.579214 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.579194 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-credential-socket\") pod \"827f550b-a8d9-4074-b34d-551ae660d799\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " Oct 31 00:50:24.579315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.579264 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-certs" pod "827f550b-a8d9-4074-b34d-551ae660d799" (UID: "827f550b-a8d9-4074-b34d-551ae660d799"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.579315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.579287 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-data\") pod \"827f550b-a8d9-4074-b34d-551ae660d799\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " Oct 31 00:50:24.579424 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.579328 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827f550b-a8d9-4074-b34d-551ae660d799-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827f550b-a8d9-4074-b34d-551ae660d799-istiod-ca-cert\") pod \"827f550b-a8d9-4074-b34d-551ae660d799\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " Oct 31 00:50:24.579503 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.579469 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-socket\") pod \"827f550b-a8d9-4074-b34d-551ae660d799\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " Oct 31 00:50:24.579503 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.579500 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-istio-token\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-istio-token\") pod \"827f550b-a8d9-4074-b34d-551ae660d799\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " Oct 31 00:50:24.579610 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.579533 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-envoy\") pod \"827f550b-a8d9-4074-b34d-551ae660d799\" (UID: \"827f550b-a8d9-4074-b34d-551ae660d799\") " Oct 31 00:50:24.580866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.580101 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-credential-socket" pod "827f550b-a8d9-4074-b34d-551ae660d799" (UID: "827f550b-a8d9-4074-b34d-551ae660d799"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.580866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.580404 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-data" pod "827f550b-a8d9-4074-b34d-551ae660d799" (UID: "827f550b-a8d9-4074-b34d-551ae660d799"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.580866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.580478 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-socket" pod "827f550b-a8d9-4074-b34d-551ae660d799" (UID: "827f550b-a8d9-4074-b34d-551ae660d799"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.580866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.580547 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.580866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.580569 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.583285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.581927 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827f550b-a8d9-4074-b34d-551ae660d799-istiod-ca-cert" pod "827f550b-a8d9-4074-b34d-551ae660d799" (UID: "827f550b-a8d9-4074-b34d-551ae660d799"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:24.583285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.582231 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/827f550b-a8d9-4074-b34d-551ae660d799-istio-podinfo" pod "827f550b-a8d9-4074-b34d-551ae660d799" (UID: "827f550b-a8d9-4074-b34d-551ae660d799"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:24.585350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.585307 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-kube-api-access-46zjr" pod "827f550b-a8d9-4074-b34d-551ae660d799" (UID: "827f550b-a8d9-4074-b34d-551ae660d799"). InnerVolumeSpecName "kube-api-access-46zjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:24.586314 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.586275 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-istio-token" pod "827f550b-a8d9-4074-b34d-551ae660d799" (UID: "827f550b-a8d9-4074-b34d-551ae660d799"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:24.587339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.587312 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-envoy" pod "827f550b-a8d9-4074-b34d-551ae660d799" (UID: "827f550b-a8d9-4074-b34d-551ae660d799"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.683032 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.682989 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zjr\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-kube-api-access-46zjr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.683032 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.683034 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.683276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.683050 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827f550b-a8d9-4074-b34d-551ae660d799-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.683276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.683062 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.683276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.683075 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827f550b-a8d9-4074-b34d-551ae660d799-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.683276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.683089 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827f550b-a8d9-4074-b34d-551ae660d799-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.683276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.683101 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827f550b-a8d9-4074-b34d-551ae660d799-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:24.890446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.890408 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:50:24.990318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.990281 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-credential-socket\") pod \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " Oct 31 00:50:24.990536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.990517 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-socket\") pod \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " Oct 31 00:50:24.990624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.990555 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-envoy\") pod \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " Oct 31 00:50:24.990624 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.990599 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istiod-ca-cert\") pod \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " Oct 31 00:50:24.990736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.990647 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-data\") pod \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " Oct 31 00:50:24.990795 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.990766 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-podinfo\") pod \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " Oct 31 00:50:24.990854 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.990825 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-certs\") pod \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " Oct 31 00:50:24.991011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.990993 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-kube-api-access-n4fms\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-kube-api-access-n4fms\") pod \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " Oct 31 00:50:24.991076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.991053 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-token\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-token\") pod \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\" (UID: \"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297\") " Oct 31 00:50:24.993320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.993179 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istiod-ca-cert" pod "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" (UID: "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:24.993320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.993281 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-credential-socket" pod "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" (UID: "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.995081 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.995054 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-certs" pod "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" (UID: "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.995942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.995917 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-socket" pod "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" (UID: "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.997567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.997541 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-data" pod "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" (UID: "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:24.999083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.999058 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:50:24.999440 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:24.999412 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-kube-api-access-n4fms" pod "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" (UID: "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297"). InnerVolumeSpecName "kube-api-access-n4fms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:25.003649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.003622 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:50:25.004979 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.004942 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-envoy" pod "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" (UID: "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.005412 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.005387 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-token" pod "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" (UID: "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:25.007279 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.007254 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-podinfo" pod "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" (UID: "2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:25.038560 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.038532 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:50:25.093794 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.093686 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-certs\") pod \"091d8d89-5935-4a95-a9c4-882e48110003\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " Oct 31 00:50:25.093794 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.093737 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-data\") pod \"8e8ef570-9a17-4fab-823d-896f9fa99067\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " Oct 31 00:50:25.093794 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.093776 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-envoy\") pod \"8e8ef570-9a17-4fab-823d-896f9fa99067\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " Oct 31 00:50:25.094095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.093806 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-credential-socket\") pod \"091d8d89-5935-4a95-a9c4-882e48110003\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " Oct 31 00:50:25.094095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.093855 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-certs\") pod \"04cb6d77-6b91-4cb8-8205-16567a3783dd\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " Oct 31 00:50:25.094095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.093894 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-istio-token\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-istio-token\") pod \"091d8d89-5935-4a95-a9c4-882e48110003\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " Oct 31 00:50:25.094095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.093996 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-socket\") pod \"8e8ef570-9a17-4fab-823d-896f9fa99067\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " Oct 31 00:50:25.094095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094042 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04cb6d77-6b91-4cb8-8205-16567a3783dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cb6d77-6b91-4cb8-8205-16567a3783dd-istiod-ca-cert\") pod \"04cb6d77-6b91-4cb8-8205-16567a3783dd\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " Oct 31 00:50:25.094362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094111 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-kube-api-access-zcp22\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-kube-api-access-zcp22\") pod \"091d8d89-5935-4a95-a9c4-882e48110003\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " Oct 31 00:50:25.094362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094141 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8ef570-9a17-4fab-823d-896f9fa99067-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8ef570-9a17-4fab-823d-896f9fa99067-istiod-ca-cert\") pod \"8e8ef570-9a17-4fab-823d-896f9fa99067\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " Oct 31 00:50:25.094362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094173 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-socket\") pod \"091d8d89-5935-4a95-a9c4-882e48110003\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " Oct 31 00:50:25.094362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094195 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-podinfo\") pod \"8e8ef570-9a17-4fab-823d-896f9fa99067\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " Oct 31 00:50:25.094362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094219 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-certs\") pod \"8e8ef570-9a17-4fab-823d-896f9fa99067\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " Oct 31 00:50:25.094362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094241 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-kube-api-access-pdpnh\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-kube-api-access-pdpnh\") pod \"04cb6d77-6b91-4cb8-8205-16567a3783dd\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " Oct 31 00:50:25.094362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094291 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-podinfo\") pod \"04cb6d77-6b91-4cb8-8205-16567a3783dd\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " Oct 31 00:50:25.094362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094343 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/091d8d89-5935-4a95-a9c4-882e48110003-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091d8d89-5935-4a95-a9c4-882e48110003-istio-podinfo\") pod \"091d8d89-5935-4a95-a9c4-882e48110003\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " Oct 31 00:50:25.094803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094384 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-kube-api-access-p8gb7\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-kube-api-access-p8gb7\") pod \"8e8ef570-9a17-4fab-823d-896f9fa99067\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " Oct 31 00:50:25.094803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094436 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-envoy\") pod \"091d8d89-5935-4a95-a9c4-882e48110003\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " Oct 31 00:50:25.094803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094657 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-token\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-token\") pod \"04cb6d77-6b91-4cb8-8205-16567a3783dd\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " Oct 31 00:50:25.094803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094723 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-data\") pod \"04cb6d77-6b91-4cb8-8205-16567a3783dd\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " Oct 31 00:50:25.094803 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094751 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-token\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-token\") pod \"8e8ef570-9a17-4fab-823d-896f9fa99067\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " Oct 31 00:50:25.095053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094843 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-envoy\") pod \"04cb6d77-6b91-4cb8-8205-16567a3783dd\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " Oct 31 00:50:25.095053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094841 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-data" pod "8e8ef570-9a17-4fab-823d-896f9fa99067" (UID: "8e8ef570-9a17-4fab-823d-896f9fa99067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.095053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094914 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-socket\") pod \"04cb6d77-6b91-4cb8-8205-16567a3783dd\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " Oct 31 00:50:25.095053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094964 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091d8d89-5935-4a95-a9c4-882e48110003-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091d8d89-5935-4a95-a9c4-882e48110003-istiod-ca-cert\") pod \"091d8d89-5935-4a95-a9c4-882e48110003\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " Oct 31 00:50:25.095053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.094993 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-credential-socket\") pod \"8e8ef570-9a17-4fab-823d-896f9fa99067\" (UID: \"8e8ef570-9a17-4fab-823d-896f9fa99067\") " Oct 31 00:50:25.095053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.095045 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-credential-socket\") pod \"04cb6d77-6b91-4cb8-8205-16567a3783dd\" (UID: \"04cb6d77-6b91-4cb8-8205-16567a3783dd\") " Oct 31 00:50:25.095360 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.095154 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-data\") pod \"091d8d89-5935-4a95-a9c4-882e48110003\" (UID: \"091d8d89-5935-4a95-a9c4-882e48110003\") " Oct 31 00:50:25.095997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.095964 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-certs" pod "04cb6d77-6b91-4cb8-8205-16567a3783dd" (UID: "04cb6d77-6b91-4cb8-8205-16567a3783dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.096249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096193 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-socket" pod "8e8ef570-9a17-4fab-823d-896f9fa99067" (UID: "8e8ef570-9a17-4fab-823d-896f9fa99067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.096656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096621 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.096743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096664 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.096743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096682 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.096743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096696 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.096743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096709 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.096743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096722 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.096743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096735 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.097173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096748 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.097173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096763 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.097173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096777 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4fms\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-kube-api-access-n4fms\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.097173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096791 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.097173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.096804 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.097437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.097369 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-envoy" pod "8e8ef570-9a17-4fab-823d-896f9fa99067" (UID: "8e8ef570-9a17-4fab-823d-896f9fa99067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.097846 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.097820 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8ef570-9a17-4fab-823d-896f9fa99067-istiod-ca-cert" pod "8e8ef570-9a17-4fab-823d-896f9fa99067" (UID: "8e8ef570-9a17-4fab-823d-896f9fa99067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:25.098144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.098121 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-data" pod "04cb6d77-6b91-4cb8-8205-16567a3783dd" (UID: "04cb6d77-6b91-4cb8-8205-16567a3783dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.122102 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.122056 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-kube-api-access-pdpnh" pod "04cb6d77-6b91-4cb8-8205-16567a3783dd" (UID: "04cb6d77-6b91-4cb8-8205-16567a3783dd"). InnerVolumeSpecName "kube-api-access-pdpnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:25.122611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.122581 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cb6d77-6b91-4cb8-8205-16567a3783dd-istiod-ca-cert" pod "04cb6d77-6b91-4cb8-8205-16567a3783dd" (UID: "04cb6d77-6b91-4cb8-8205-16567a3783dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:25.124657 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.123054 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-credential-socket" pod "091d8d89-5935-4a95-a9c4-882e48110003" (UID: "091d8d89-5935-4a95-a9c4-882e48110003"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.124657 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.123302 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-credential-socket" pod "8e8ef570-9a17-4fab-823d-896f9fa99067" (UID: "8e8ef570-9a17-4fab-823d-896f9fa99067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.124657 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.123911 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-certs" pod "8e8ef570-9a17-4fab-823d-896f9fa99067" (UID: "8e8ef570-9a17-4fab-823d-896f9fa99067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.124657 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.124542 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-data" pod "091d8d89-5935-4a95-a9c4-882e48110003" (UID: "091d8d89-5935-4a95-a9c4-882e48110003"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.124657 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.124628 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-socket" pod "091d8d89-5935-4a95-a9c4-882e48110003" (UID: "091d8d89-5935-4a95-a9c4-882e48110003"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.125723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.125684 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091d8d89-5935-4a95-a9c4-882e48110003-istiod-ca-cert" pod "091d8d89-5935-4a95-a9c4-882e48110003" (UID: "091d8d89-5935-4a95-a9c4-882e48110003"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:25.125927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.125894 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-certs" pod "091d8d89-5935-4a95-a9c4-882e48110003" (UID: "091d8d89-5935-4a95-a9c4-882e48110003"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.127121 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.127055 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-podinfo" pod "8e8ef570-9a17-4fab-823d-896f9fa99067" (UID: "8e8ef570-9a17-4fab-823d-896f9fa99067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:25.128561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.127113 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-kube-api-access-p8gb7" pod "8e8ef570-9a17-4fab-823d-896f9fa99067" (UID: "8e8ef570-9a17-4fab-823d-896f9fa99067"). InnerVolumeSpecName "kube-api-access-p8gb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:25.128670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.127525 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-kube-api-access-zcp22" pod "091d8d89-5935-4a95-a9c4-882e48110003" (UID: "091d8d89-5935-4a95-a9c4-882e48110003"). InnerVolumeSpecName "kube-api-access-zcp22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:25.128670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.127622 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-envoy" pod "091d8d89-5935-4a95-a9c4-882e48110003" (UID: "091d8d89-5935-4a95-a9c4-882e48110003"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.128670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.127754 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-istio-token" pod "091d8d89-5935-4a95-a9c4-882e48110003" (UID: "091d8d89-5935-4a95-a9c4-882e48110003"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:25.129065 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.129039 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/091d8d89-5935-4a95-a9c4-882e48110003-istio-podinfo" pod "091d8d89-5935-4a95-a9c4-882e48110003" (UID: "091d8d89-5935-4a95-a9c4-882e48110003"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:25.130632 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.130606 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-token" pod "8e8ef570-9a17-4fab-823d-896f9fa99067" (UID: "8e8ef570-9a17-4fab-823d-896f9fa99067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:25.145750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.145696 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-socket" pod "04cb6d77-6b91-4cb8-8205-16567a3783dd" (UID: "04cb6d77-6b91-4cb8-8205-16567a3783dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.147288 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.147253 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-podinfo" pod "04cb6d77-6b91-4cb8-8205-16567a3783dd" (UID: "04cb6d77-6b91-4cb8-8205-16567a3783dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:25.148200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.148172 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-envoy" pod "04cb6d77-6b91-4cb8-8205-16567a3783dd" (UID: "04cb6d77-6b91-4cb8-8205-16567a3783dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.148625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.148595 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-token" pod "04cb6d77-6b91-4cb8-8205-16567a3783dd" (UID: "04cb6d77-6b91-4cb8-8205-16567a3783dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:25.149471 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.149416 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-credential-socket" pod "04cb6d77-6b91-4cb8-8205-16567a3783dd" (UID: "04cb6d77-6b91-4cb8-8205-16567a3783dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:25.199253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199215 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cb6d77-6b91-4cb8-8205-16567a3783dd-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199245 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcp22\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-kube-api-access-zcp22\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199255 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8ef570-9a17-4fab-823d-896f9fa99067-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199272 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199284 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199295 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199304 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdpnh\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-kube-api-access-pdpnh\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199312 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199320 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091d8d89-5935-4a95-a9c4-882e48110003-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199328 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gb7\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-kube-api-access-p8gb7\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199337 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199345 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199352 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199362 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199374 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199388 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199396 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091d8d89-5935-4a95-a9c4-882e48110003-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199405 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199413 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cb6d77-6b91-4cb8-8205-16567a3783dd-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199422 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199431 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199440 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8ef570-9a17-4fab-823d-896f9fa99067-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199497 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091d8d89-5935-4a95-a9c4-882e48110003-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.199616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.199509 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091d8d89-5935-4a95-a9c4-882e48110003-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:25.241372 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.241339 5978 generic.go:356] "Generic (PLEG): container finished" podID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerID="96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb" exitCode=0 Oct 31 00:50:25.241571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.241392 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" Oct 31 00:50:25.241571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.241423 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" event={"ID":"04cb6d77-6b91-4cb8-8205-16567a3783dd","Type":"ContainerDied","Data":"96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb"} Oct 31 00:50:25.241571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.241539 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp" event={"ID":"04cb6d77-6b91-4cb8-8205-16567a3783dd","Type":"ContainerDied","Data":"f8e824a31ea047c07fdc8ec8ac98ffd35b0e01ac63ae7298df6e09683257ff0e"} Oct 31 00:50:25.241571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.241567 5978 scope.go:122] "RemoveContainer" containerID="96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb" Oct 31 00:50:25.243024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.242993 5978 generic.go:356] "Generic (PLEG): container finished" podID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerID="4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7" exitCode=0 Oct 31 00:50:25.243119 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.243040 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" Oct 31 00:50:25.243119 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.243073 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" event={"ID":"8e8ef570-9a17-4fab-823d-896f9fa99067","Type":"ContainerDied","Data":"4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7"} Oct 31 00:50:25.243232 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.243119 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85" event={"ID":"8e8ef570-9a17-4fab-823d-896f9fa99067","Type":"ContainerDied","Data":"e9b2d12c83a8ff6ed5c5a0a44e7fe90081280d7353794831f140a6a686460d7a"} Oct 31 00:50:25.244991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.244967 5978 generic.go:356] "Generic (PLEG): container finished" podID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerID="3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11" exitCode=0 Oct 31 00:50:25.245120 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.245046 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" Oct 31 00:50:25.245120 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.245087 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" event={"ID":"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297","Type":"ContainerDied","Data":"3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11"} Oct 31 00:50:25.245234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.245133 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l" event={"ID":"2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297","Type":"ContainerDied","Data":"96f74ee3cc99d5f74608e114f84df3dc6e5179e7a27db4ee6dba0f10621103a5"} Oct 31 00:50:25.246941 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.246920 5978 generic.go:356] "Generic (PLEG): container finished" podID="827f550b-a8d9-4074-b34d-551ae660d799" containerID="ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c" exitCode=0 Oct 31 00:50:25.247043 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.247018 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" Oct 31 00:50:25.247043 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.247031 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" event={"ID":"827f550b-a8d9-4074-b34d-551ae660d799","Type":"ContainerDied","Data":"ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c"} Oct 31 00:50:25.247215 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.247072 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk" event={"ID":"827f550b-a8d9-4074-b34d-551ae660d799","Type":"ContainerDied","Data":"2b35cd2991f37471dd122d75c679f639a3e30e5710d0af207359a9f963b95210"} Oct 31 00:50:25.249646 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.249619 5978 generic.go:356] "Generic (PLEG): container finished" podID="091d8d89-5935-4a95-a9c4-882e48110003" containerID="a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0" exitCode=0 Oct 31 00:50:25.249776 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.249668 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" Oct 31 00:50:25.249776 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.249721 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" event={"ID":"091d8d89-5935-4a95-a9c4-882e48110003","Type":"ContainerDied","Data":"a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0"} Oct 31 00:50:25.249776 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.249763 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb" event={"ID":"091d8d89-5935-4a95-a9c4-882e48110003","Type":"ContainerDied","Data":"79f77b52f0322568f7c9d198ac31532de3027a01d285d4565a982c86a19a6029"} Oct 31 00:50:25.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.271644 5978 scope.go:122] "RemoveContainer" containerID="96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb" Oct 31 00:50:25.272179 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:50:25.272144 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb\": container with ID starting with 96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb not found: ID does not exist" containerID="96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb" Oct 31 00:50:25.272339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.272191 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb"} err="failed to get container status \"96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb\": rpc error: code = NotFound desc = could not find container \"96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb\": container with ID starting with 96011b1f00d231295361f2459f597f316b876838ff9f9c69cf196ad2f18911eb not found: ID does not exist" Oct 31 00:50:25.272339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.272243 5978 scope.go:122] "RemoveContainer" containerID="4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7" Oct 31 00:50:25.318420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.318391 5978 scope.go:122] "RemoveContainer" containerID="4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7" Oct 31 00:50:25.318836 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:50:25.318809 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7\": container with ID starting with 4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7 not found: ID does not exist" containerID="4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7" Oct 31 00:50:25.318950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.318843 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7"} err="failed to get container status \"4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7\": rpc error: code = NotFound desc = could not find container \"4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7\": container with ID starting with 4d5c3fe02fbceacf3186c9e790a01908aaf886d5f14d77780a99c4550ae6f8d7 not found: ID does not exist" Oct 31 00:50:25.318950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.318867 5978 scope.go:122] "RemoveContainer" containerID="3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11" Oct 31 00:50:25.332335 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.332310 5978 scope.go:122] "RemoveContainer" containerID="3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11" Oct 31 00:50:25.332708 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:50:25.332686 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11\": container with ID starting with 3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11 not found: ID does not exist" containerID="3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11" Oct 31 00:50:25.332797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.332716 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11"} err="failed to get container status \"3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11\": rpc error: code = NotFound desc = could not find container \"3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11\": container with ID starting with 3b9027260900398dad5bf2b3fc075fe1cfaccbeec27ea952014771532317ea11 not found: ID does not exist" Oct 31 00:50:25.332797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.332734 5978 scope.go:122] "RemoveContainer" containerID="ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c" Oct 31 00:50:25.345613 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.345584 5978 scope.go:122] "RemoveContainer" containerID="ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c" Oct 31 00:50:25.345974 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:50:25.345946 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c\": container with ID starting with ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c not found: ID does not exist" containerID="ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c" Oct 31 00:50:25.346044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.345986 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c"} err="failed to get container status \"ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c\": rpc error: code = NotFound desc = could not find container \"ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c\": container with ID starting with ba07a3f40e3cfe41388bbae741be4bede99a9beb841d161ae05b4e1b47d6216c not found: ID does not exist" Oct 31 00:50:25.346044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.346007 5978 scope.go:122] "RemoveContainer" containerID="a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0" Oct 31 00:50:25.360404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.359418 5978 scope.go:122] "RemoveContainer" containerID="a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0" Oct 31 00:50:25.360404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.359558 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk"] Oct 31 00:50:25.360404 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:50:25.359894 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0\": container with ID starting with a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0 not found: ID does not exist" containerID="a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0" Oct 31 00:50:25.360404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.359925 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0"} err="failed to get container status \"a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0\": rpc error: code = NotFound desc = could not find container \"a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0\": container with ID starting with a0e292177e3b097a98500380ead66267b0b632c2d3a0b1ccc8800365090fd9e0 not found: ID does not exist" Oct 31 00:50:25.364464 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.364427 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-tfpjk"] Oct 31 00:50:25.395223 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.395191 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp"] Oct 31 00:50:25.407905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.407870 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-xgzzp"] Oct 31 00:50:25.437046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.437013 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb"] Oct 31 00:50:25.447196 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.447160 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-mctcb"] Oct 31 00:50:25.478927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.478883 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l"] Oct 31 00:50:25.483868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.483832 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-jgr7l"] Oct 31 00:50:25.486169 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.486135 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:25.515102 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.515069 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85"] Oct 31 00:50:25.519419 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:25.519387 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9e377-openshift-default-bbb7b4499-2wn85"] Oct 31 00:50:26.431227 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:26.431191 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" path="/var/lib/kubelet/pods/04cb6d77-6b91-4cb8-8205-16567a3783dd/volumes" Oct 31 00:50:26.433070 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:26.433050 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091d8d89-5935-4a95-a9c4-882e48110003" path="/var/lib/kubelet/pods/091d8d89-5935-4a95-a9c4-882e48110003/volumes" Oct 31 00:50:26.433588 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:26.433573 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" path="/var/lib/kubelet/pods/2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297/volumes" Oct 31 00:50:26.434349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:26.434334 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827f550b-a8d9-4074-b34d-551ae660d799" path="/var/lib/kubelet/pods/827f550b-a8d9-4074-b34d-551ae660d799/volumes" Oct 31 00:50:26.434845 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:26.434831 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" path="/var/lib/kubelet/pods/8e8ef570-9a17-4fab-823d-896f9fa99067/volumes" Oct 31 00:50:26.659832 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:26.659798 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:27.432475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:27.428353 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:27.579558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:27.579522 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:27.777538 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:27.775028 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:29.851198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:29.851156 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:32.131478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.130536 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb"] Oct 31 00:50:32.190477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.189822 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96814a5d-6vtwb"] Oct 31 00:50:32.360771 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.360737 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:32.399818 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.399715 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.422918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.408623 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.549477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.545103 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072b4b37-7db9-4dce-a3b7-41eb4a00022d" path="/var/lib/kubelet/pods/072b4b37-7db9-4dce-a3b7-41eb4a00022d/volumes" Oct 31 00:50:32.619712 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.619667 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.644280 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.644244 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:32.908479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.907608 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.908701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.908633 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.992097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.990984 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.992514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.992477 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:32.995695 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:32.995667 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Oct 31 00:50:33.084429 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:33.084382 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:33.210849 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:50:33.210731 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:50:35.010235 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:35.010168 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:36.705247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:36.705197 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:38.513417 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:38.513369 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:40.238475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:40.237070 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:41.865342 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:41.865305 5978 generic.go:356] "Generic (PLEG): container finished" podID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerID="433a159cdd9a7ab4e0cb271cc4f02c1e063610ec787d9d564137350efff32599" exitCode=0 Oct 31 00:50:41.865901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:41.865384 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" event={"ID":"3a043e49-6ec4-47b9-8323-d49ee320d95c","Type":"ContainerDied","Data":"433a159cdd9a7ab4e0cb271cc4f02c1e063610ec787d9d564137350efff32599"} Oct 31 00:50:41.865980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:41.865963 5978 scope.go:122] "RemoveContainer" containerID="433a159cdd9a7ab4e0cb271cc4f02c1e063610ec787d9d564137350efff32599" Oct 31 00:50:41.868399 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:41.868363 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:41.868558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:41.868472 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:41.868558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:41.868530 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:42.020315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:42.020268 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:42.877211 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:42.877179 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" event={"ID":"3a043e49-6ec4-47b9-8323-d49ee320d95c","Type":"ContainerStarted","Data":"13802e888d9ebe99cf0d832460df845156d9f120864a29ebebcc42a376d39990"} Oct 31 00:50:42.913040 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:42.913009 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:50:42.914768 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:42.914741 5978 patch_prober.go:29] interesting pod/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:50:42.914899 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:42.914788 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:50:43.865570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:43.865533 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:43.939480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:43.935773 5978 patch_prober.go:29] interesting pod/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:50:43.940160 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:43.940102 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:50:44.103016 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:44.102980 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:50:44.922483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:44.915992 5978 patch_prober.go:29] interesting pod/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:50:44.922483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:44.916061 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:50:45.005011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:45.004924 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:45.156626 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:45.156586 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:45.871327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:45.871290 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:45.916482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:45.915358 5978 patch_prober.go:29] interesting pod/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:50:45.916482 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:45.915475 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:50:46.077328 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:46.077296 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.233368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:46.233276 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.450964 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:46.450932 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.922478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:46.918903 5978 patch_prober.go:29] interesting pod/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:50:46.922478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:46.918985 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:50:47.197439 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:47.197356 5978 scope.go:122] "RemoveContainer" containerID="c067103e2554a4769cbf8c82be6e1d836737e7de93ca1349fd813965113a97bb" Oct 31 00:50:47.209480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:47.209448 5978 scope.go:122] "RemoveContainer" containerID="f798b374929cb589481730e91b03c954731ef65c896b35991f5b26d41e8364b4" Oct 31 00:50:47.248483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:47.248463 5978 scope.go:122] "RemoveContainer" containerID="af9790a3dae89d6493a2b0eaf5feb76387b4c5cbb439cef84eebe63f85fe9422" Oct 31 00:50:47.260632 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:47.260610 5978 scope.go:122] "RemoveContainer" containerID="beca1ccadc12bc772cd8ef0f29f587c0c587f9f482ce3f04c9061c0e3fcc8db8" Oct 31 00:50:47.552435 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:47.552339 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:47.794652 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:47.794613 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:47.930476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:47.926886 5978 patch_prober.go:29] interesting pod/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:50:47.930476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:47.926967 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:50:49.043253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:49.043208 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:50:49.044709 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:49.044680 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:50:49.399590 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:49.399553 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9"] Oct 31 00:50:49.399808 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:49.399783 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" containerID="cri-o://d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3" gracePeriod=30 Oct 31 00:50:49.923404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:49.923367 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:51.640569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:51.640531 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:53.319527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:53.319493 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:54.694104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.694079 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:50:54.830366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.830302 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-data\") pod \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " Oct 31 00:50:54.830646 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.830399 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-kube-api-access-sbmf8\") pod \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " Oct 31 00:50:54.830646 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.830478 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-envoy\") pod \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " Oct 31 00:50:54.831661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.830877 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-data" pod "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" (UID: "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.831661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.831350 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-podinfo\") pod \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " Oct 31 00:50:54.831661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.831411 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-socket\") pod \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " Oct 31 00:50:54.831661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.831442 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-token\") pod \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " Oct 31 00:50:54.831661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.831524 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-credential-socket\") pod \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " Oct 31 00:50:54.831661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.831589 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istiod-ca-cert\") pod \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " Oct 31 00:50:54.831995 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.831714 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-certs\") pod \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\" (UID: \"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1\") " Oct 31 00:50:54.832682 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.832656 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:54.832950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.832918 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-credential-socket" pod "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" (UID: "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.833073 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.832993 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-socket" pod "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" (UID: "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.833213 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.833187 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istiod-ca-cert" pod "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" (UID: "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.833496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.833436 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-kube-api-access-sbmf8" pod "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" (UID: "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1"). InnerVolumeSpecName "kube-api-access-sbmf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.833583 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.833516 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-certs" pod "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" (UID: "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.833630 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.833590 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-envoy" pod "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" (UID: "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.834695 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.834673 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-token" pod "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" (UID: "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.835338 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.835326 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-podinfo" pod "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" (UID: "1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:54.894654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.894619 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:54.936253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.936118 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:54.936253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.936152 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-kube-api-access-sbmf8\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:54.936253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.936167 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:54.936253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.936181 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:54.936253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.936193 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:54.936253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.936206 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:54.936253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.936220 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:54.936253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:54.936232 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:50:55.053198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.053160 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:55.062781 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.062750 5978 generic.go:356] "Generic (PLEG): container finished" podID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerID="d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3" exitCode=0 Oct 31 00:50:55.062963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.062800 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" Oct 31 00:50:55.062963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.062837 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" event={"ID":"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1","Type":"ContainerDied","Data":"d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3"} Oct 31 00:50:55.062963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.062898 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9" event={"ID":"1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1","Type":"ContainerDied","Data":"4ea952e210bce7b936e57af552a3b43cee7cd51c9e370a4ae4b0181cfe6339b8"} Oct 31 00:50:55.062963 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.062926 5978 scope.go:122] "RemoveContainer" containerID="d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3" Oct 31 00:50:55.079071 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.079048 5978 scope.go:122] "RemoveContainer" containerID="d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3" Oct 31 00:50:55.079437 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:50:55.079416 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3\": container with ID starting with d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3 not found: ID does not exist" containerID="d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3" Oct 31 00:50:55.079587 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.079445 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3"} err="failed to get container status \"d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3\": rpc error: code = NotFound desc = could not find container \"d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3\": container with ID starting with d63e0d10ee89de9d51676806bb4e03d3f49074c2ea457b56ea78951d546fdad3 not found: ID does not exist" Oct 31 00:50:55.116524 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.115969 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9"] Oct 31 00:50:55.119315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.119284 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-pjgr9"] Oct 31 00:50:55.277680 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:55.277582 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:56.432578 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:56.432542 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" path="/var/lib/kubelet/pods/1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1/volumes" Oct 31 00:50:56.896001 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:56.895934 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:58.032871 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:58.032837 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf"] Oct 31 00:50:58.069745 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:58.069690 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2"] Oct 31 00:50:58.217172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:58.217137 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:58.362736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:58.362699 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:58.910433 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:50:58.910391 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:00.720966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:00.720922 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:02.370010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:02.369972 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:03.033241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.033192 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" podUID="26e6fb79-dceb-45cd-becf-978886405174" containerName="payload-processing" containerID="cri-o://fd029dd31a4051666a0750759f2e83e042b54de2fab006e302d0a04b8d8357f2" gracePeriod=25 Oct 31 00:51:03.070160 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.070119 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" podUID="3175f116-c972-47f5-bc69-1f3ee3fd2ac7" containerName="payload-pre-processing" containerID="cri-o://c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922" gracePeriod=25 Oct 31 00:51:03.179523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.179494 5978 generic.go:356] "Generic (PLEG): container finished" podID="26e6fb79-dceb-45cd-becf-978886405174" containerID="fd029dd31a4051666a0750759f2e83e042b54de2fab006e302d0a04b8d8357f2" exitCode=0 Oct 31 00:51:03.179662 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.179631 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" event={"ID":"26e6fb79-dceb-45cd-becf-978886405174","Type":"ContainerDied","Data":"fd029dd31a4051666a0750759f2e83e042b54de2fab006e302d0a04b8d8357f2"} Oct 31 00:51:03.188495 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:51:03.188415 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:51:03.398203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.398176 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:51:03.427826 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.427801 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:51:03.527421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.527377 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e6fb79-dceb-45cd-becf-978886405174-kube-api-access-2fgv8\" (UniqueName: \"kubernetes.io/projected/26e6fb79-dceb-45cd-becf-978886405174-kube-api-access-2fgv8\") pod \"26e6fb79-dceb-45cd-becf-978886405174\" (UID: \"26e6fb79-dceb-45cd-becf-978886405174\") " Oct 31 00:51:03.527641 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.527541 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26e6fb79-dceb-45cd-becf-978886405174-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e6fb79-dceb-45cd-becf-978886405174-plugins-config-volume\") pod \"26e6fb79-dceb-45cd-becf-978886405174\" (UID: \"26e6fb79-dceb-45cd-becf-978886405174\") " Oct 31 00:51:03.527715 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.527642 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-kube-api-access-9g949\" (UniqueName: \"kubernetes.io/projected/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-kube-api-access-9g949\") pod \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\" (UID: \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\") " Oct 31 00:51:03.527888 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.527868 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-plugins-config-volume\") pod \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\" (UID: \"3175f116-c972-47f5-bc69-1f3ee3fd2ac7\") " Oct 31 00:51:03.528052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.528006 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e6fb79-dceb-45cd-becf-978886405174-plugins-config-volume" pod "26e6fb79-dceb-45cd-becf-978886405174" (UID: "26e6fb79-dceb-45cd-becf-978886405174"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:03.528257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.528233 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-plugins-config-volume" pod "3175f116-c972-47f5-bc69-1f3ee3fd2ac7" (UID: "3175f116-c972-47f5-bc69-1f3ee3fd2ac7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:03.528978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.528956 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:03.528978 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.528977 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26e6fb79-dceb-45cd-becf-978886405174-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:03.530574 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.530547 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-kube-api-access-9g949" pod "3175f116-c972-47f5-bc69-1f3ee3fd2ac7" (UID: "3175f116-c972-47f5-bc69-1f3ee3fd2ac7"). InnerVolumeSpecName "kube-api-access-9g949". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:03.530707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.530642 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e6fb79-dceb-45cd-becf-978886405174-kube-api-access-2fgv8" pod "26e6fb79-dceb-45cd-becf-978886405174" (UID: "26e6fb79-dceb-45cd-becf-978886405174"). InnerVolumeSpecName "kube-api-access-2fgv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:03.630176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.630136 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g949\" (UniqueName: \"kubernetes.io/projected/3175f116-c972-47f5-bc69-1f3ee3fd2ac7-kube-api-access-9g949\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:03.630176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.630171 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fgv8\" (UniqueName: \"kubernetes.io/projected/26e6fb79-dceb-45cd-becf-978886405174-kube-api-access-2fgv8\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:03.989002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:03.988921 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:04.189343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.189303 5978 generic.go:356] "Generic (PLEG): container finished" podID="3175f116-c972-47f5-bc69-1f3ee3fd2ac7" containerID="c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922" exitCode=0 Oct 31 00:51:04.189584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.189357 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" Oct 31 00:51:04.189584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.189387 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" event={"ID":"3175f116-c972-47f5-bc69-1f3ee3fd2ac7","Type":"ContainerDied","Data":"c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922"} Oct 31 00:51:04.189584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.189468 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2" event={"ID":"3175f116-c972-47f5-bc69-1f3ee3fd2ac7","Type":"ContainerDied","Data":"04af22c303c31d5e14b17e767b03dae0850ba82092e1980bd36ad7fd6d5c0d46"} Oct 31 00:51:04.189584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.189499 5978 scope.go:122] "RemoveContainer" containerID="c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922" Oct 31 00:51:04.191236 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.191218 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" Oct 31 00:51:04.191332 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.191267 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf" event={"ID":"26e6fb79-dceb-45cd-becf-978886405174","Type":"ContainerDied","Data":"94ac96a57dee3624bb7358da214b7ba0de153753597a0354f1bfb5b296b39912"} Oct 31 00:51:04.202952 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.202935 5978 scope.go:122] "RemoveContainer" containerID="c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922" Oct 31 00:51:04.203278 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:51:04.203260 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922\": container with ID starting with c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922 not found: ID does not exist" containerID="c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922" Oct 31 00:51:04.203322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.203287 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922"} err="failed to get container status \"c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922\": rpc error: code = NotFound desc = could not find container \"c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922\": container with ID starting with c8a8f953686073db3d4970ad6bb5c86cf0b6366a6cab78bdec1e5135b6f35922 not found: ID does not exist" Oct 31 00:51:04.203322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.203308 5978 scope.go:122] "RemoveContainer" containerID="fd029dd31a4051666a0750759f2e83e042b54de2fab006e302d0a04b8d8357f2" Oct 31 00:51:04.241389 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.241311 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2"] Oct 31 00:51:04.251383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.251336 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e166ca-5c477786c9-79vt2"] Oct 31 00:51:04.279603 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.279570 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf"] Oct 31 00:51:04.285906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.285872 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e166ca-7974df756c-ltngf"] Oct 31 00:51:04.427885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.427851 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26e6fb79-dceb-45cd-becf-978886405174" path="/var/lib/kubelet/pods/26e6fb79-dceb-45cd-becf-978886405174/volumes" Oct 31 00:51:04.428335 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:04.428320 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3175f116-c972-47f5-bc69-1f3ee3fd2ac7" path="/var/lib/kubelet/pods/3175f116-c972-47f5-bc69-1f3ee3fd2ac7/volumes" Oct 31 00:51:05.704665 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:05.704626 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:07.471655 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:07.471616 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:07.617862 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:07.617823 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:08.744254 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:08.744222 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75"] Oct 31 00:51:08.744750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:08.744467 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" podUID="7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" containerName="maas-api" containerID="cri-o://eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e" gracePeriod=30 Oct 31 00:51:08.915315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:08.915274 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:09.212421 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.212060 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:51:09.256827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.252937 5978 generic.go:356] "Generic (PLEG): container finished" podID="7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" containerID="eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e" exitCode=0 Oct 31 00:51:09.256827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.253027 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" event={"ID":"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02","Type":"ContainerDied","Data":"eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e"} Oct 31 00:51:09.256827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.253037 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" Oct 31 00:51:09.256827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.253084 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75" event={"ID":"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02","Type":"ContainerDied","Data":"8619a3571f077703cdbae6509131c6f6665844414f710bbf49d9d1a4590f8b88"} Oct 31 00:51:09.256827 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.253114 5978 scope.go:122] "RemoveContainer" containerID="eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e" Oct 31 00:51:09.272691 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.271692 5978 scope.go:122] "RemoveContainer" containerID="eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e" Oct 31 00:51:09.272691 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:51:09.272128 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e\": container with ID starting with eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e not found: ID does not exist" containerID="eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e" Oct 31 00:51:09.272691 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.272168 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e"} err="failed to get container status \"eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e\": rpc error: code = NotFound desc = could not find container \"eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e\": container with ID starting with eda0146ad65faf74c5af487e2608ca240bc3ccbd1465a2cb9a308728fd67677e not found: ID does not exist" Oct 31 00:51:09.282343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.280902 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-openshift-service-ca\") pod \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " Oct 31 00:51:09.282343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.280976 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-kube-api-access-hv99p\" (UniqueName: \"kubernetes.io/projected/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-kube-api-access-hv99p\") pod \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " Oct 31 00:51:09.282343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.281093 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-maas-api-tls\") pod \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " Oct 31 00:51:09.282343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.281301 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-metrics-certs\") pod \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\" (UID: \"7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02\") " Oct 31 00:51:09.286281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.286100 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-metrics-certs" pod "7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" (UID: "7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:09.286702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.286664 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-openshift-service-ca" pod "7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" (UID: "7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:09.287055 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.287019 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-kube-api-access-hv99p" pod "7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" (UID: "7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02"). InnerVolumeSpecName "kube-api-access-hv99p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:09.308813 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.308782 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-maas-api-tls" pod "7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" (UID: "7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:09.460647 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.432938 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:09.462700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.462615 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:09.462700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.462646 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:09.462700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.462661 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:09.462700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.462675 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv99p\" (UniqueName: \"kubernetes.io/projected/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02-kube-api-access-hv99p\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:09.590119 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.590080 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:09.611127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.611094 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75"] Oct 31 00:51:09.617984 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.617944 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fa2c15c1-d9ddbdddb-tcx75"] Oct 31 00:51:09.789950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:09.789857 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:10.214913 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:10.214875 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:10.428542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:10.428505 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" path="/var/lib/kubelet/pods/7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02/volumes" Oct 31 00:51:10.470411 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:10.470326 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:11.087187 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:11.087148 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:11.262475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:11.248377 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:11.407061 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:11.406955 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:11.855666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:11.855624 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:12.033474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:12.033426 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:12.181584 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:12.181436 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:14.210754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:14.210716 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.218297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:15.218265 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.530896 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:15.530805 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.284300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:16.284265 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.492476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:16.490192 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:17.232380 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:17.232341 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:17.441903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:17.441853 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:18.457275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:18.457228 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:19.104273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:19.104241 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:19.316135 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:19.316098 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:19.538642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:19.538555 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:19.721048 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:19.721009 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:20.486481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:20.484067 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:20.774115 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:20.774027 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:21.162148 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:21.159556 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:21.694608 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:21.694571 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:23.116975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:23.116933 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:25.388816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:25.388719 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:25.542592 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:25.542552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:25.689081 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:25.688990 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:27.321642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:27.321608 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:27.472283 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:27.472245 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:27.703223 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:27.703102 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:29.070491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:29.070439 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:29.744172 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:29.744140 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:29.895171 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:29.895130 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:30.148647 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:30.148610 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:30.478857 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:30.478761 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:30.703698 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:30.703652 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:31.407297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.407260 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:31.887206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.887167 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg"] Oct 31 00:51:31.902161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.902108 5978 cpu_manager.go:414] "containerMap: removing container" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" Oct 31 00:51:31.902360 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.902344 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" Oct 31 00:51:31.902508 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.902488 5978 cpu_manager.go:414] "containerMap: removing container" podUID="26e6fb79-dceb-45cd-becf-978886405174" containerName="payload-processing" Oct 31 00:51:31.902608 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.902597 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26e6fb79-dceb-45cd-becf-978886405174" containerName="payload-processing" Oct 31 00:51:31.902731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.902721 5978 cpu_manager.go:414] "containerMap: removing container" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" Oct 31 00:51:31.902807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.902797 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" Oct 31 00:51:31.902893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.902884 5978 cpu_manager.go:414] "containerMap: removing container" podUID="be61c2cf-6d4a-40ab-a95b-e4413a1394a7" containerName="payload-pre-processing" Oct 31 00:51:31.902968 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.902957 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be61c2cf-6d4a-40ab-a95b-e4413a1394a7" containerName="payload-pre-processing" Oct 31 00:51:31.903048 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903039 5978 cpu_manager.go:414] "containerMap: removing container" podUID="7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" containerName="maas-api" Oct 31 00:51:31.903119 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903110 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" containerName="maas-api" Oct 31 00:51:31.903197 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903189 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" Oct 31 00:51:31.903276 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903266 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" Oct 31 00:51:31.903359 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903350 5978 cpu_manager.go:414] "containerMap: removing container" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" Oct 31 00:51:31.903431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903422 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" Oct 31 00:51:31.903547 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903535 5978 cpu_manager.go:414] "containerMap: removing container" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" Oct 31 00:51:31.903620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903610 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" Oct 31 00:51:31.903708 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903698 5978 cpu_manager.go:414] "containerMap: removing container" podUID="3175f116-c972-47f5-bc69-1f3ee3fd2ac7" containerName="payload-pre-processing" Oct 31 00:51:31.903776 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903766 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3175f116-c972-47f5-bc69-1f3ee3fd2ac7" containerName="payload-pre-processing" Oct 31 00:51:31.903875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903851 5978 cpu_manager.go:414] "containerMap: removing container" podUID="740368e2-d7f8-41ec-ac01-3e5b9b2fa707" containerName="payload-processing" Oct 31 00:51:31.903875 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903865 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740368e2-d7f8-41ec-ac01-3e5b9b2fa707" containerName="payload-processing" Oct 31 00:51:31.904066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903884 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" Oct 31 00:51:31.904066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903892 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" Oct 31 00:51:31.904066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903903 5978 cpu_manager.go:414] "containerMap: removing container" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" Oct 31 00:51:31.904066 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.903909 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904178 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acdf3b2-d3c6-4c7f-aab7-3ce68c0ba297" containerName="istio-proxy" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904193 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="26e6fb79-dceb-45cd-becf-978886405174" containerName="payload-processing" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904204 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="827f550b-a8d9-4074-b34d-551ae660d799" containerName="istio-proxy" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904214 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="be61c2cf-6d4a-40ab-a95b-e4413a1394a7" containerName="payload-pre-processing" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904228 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1f6b59-cef6-4ed9-a295-0ea82a5ebb02" containerName="maas-api" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904238 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8ef570-9a17-4fab-823d-896f9fa99067" containerName="istio-proxy" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904247 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5e4e98-59e4-4995-bb6a-4a6c0c2296f1" containerName="istio-proxy" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904260 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cb6d77-6b91-4cb8-8205-16567a3783dd" containerName="istio-proxy" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904270 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="3175f116-c972-47f5-bc69-1f3ee3fd2ac7" containerName="payload-pre-processing" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904278 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="740368e2-d7f8-41ec-ac01-3e5b9b2fa707" containerName="payload-processing" Oct 31 00:51:31.904287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904289 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7474b67-7515-46f1-9b80-09c2353018a0" containerName="istio-proxy" Oct 31 00:51:31.904920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.904299 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="091d8d89-5935-4a95-a9c4-882e48110003" containerName="istio-proxy" Oct 31 00:51:31.910294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.910275 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:31.930423 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.930399 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-82d887-openshift-default-dockercfg-zdn5c\"" Oct 31 00:51:31.931493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:31.931444 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg"] Oct 31 00:51:32.003447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.003389 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:32.092509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.081982 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.103665 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.103623 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.123709 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.123666 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.123977 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.123959 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.124099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.124084 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.124389 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.124373 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.124574 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.124555 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8s4s\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-kube-api-access-z8s4s\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.124774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.124754 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.124905 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.124892 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.188007 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.187858 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:32.233834 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.233797 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.234567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.234542 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8s4s\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-kube-api-access-z8s4s\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.257099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.257068 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.257377 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.257355 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.257573 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.257556 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.257819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.257802 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.258109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.258085 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.276975 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.276944 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.277211 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.258773 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.277710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.258860 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.277852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.259827 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.277938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.258625 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.278033 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.277333 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.278479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.277655 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.324474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.316997 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.324474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.317616 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8s4s\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-kube-api-access-z8s4s\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.324474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.318129 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.341431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.341269 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:32.347481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.342265 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.533609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.533518 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:32.852778 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.852751 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg"] Oct 31 00:51:32.853792 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:51:32.853767 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6314229_18f2_4fbf_b206_0e7e36ac4cdd.slice/crio-934c80aced48c5c07fc3dcd4d2ec148388396a8b106796335c6e84087073956f WatchSource:0}: Error finding container 934c80aced48c5c07fc3dcd4d2ec148388396a8b106796335c6e84087073956f: Status 404 returned error can't find the container with id 934c80aced48c5c07fc3dcd4d2ec148388396a8b106796335c6e84087073956f Oct 31 00:51:32.856761 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.856724 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:51:32.856852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.856813 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:51:32.856852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:32.856843 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:51:33.184945 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:51:33.184853 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:51:33.490585 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:33.490486 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" event={"ID":"d6314229-18f2-4fbf-b206-0e7e36ac4cdd","Type":"ContainerStarted","Data":"6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386"} Oct 31 00:51:33.490585 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:33.490566 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" event={"ID":"d6314229-18f2-4fbf-b206-0e7e36ac4cdd","Type":"ContainerStarted","Data":"934c80aced48c5c07fc3dcd4d2ec148388396a8b106796335c6e84087073956f"} Oct 31 00:51:33.533860 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:33.533825 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:33.536342 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:33.536283 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podStartSLOduration=2.536264986 podStartE2EDuration="2.536264986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:33.530672639 +0000 UTC m=+1911.228786408" watchObservedRunningTime="2026-10-31 00:51:33.536264986 +0000 UTC m=+1911.234378751" Oct 31 00:51:33.540406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:33.540383 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:34.390477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:34.384636 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:34.500291 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:34.500259 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:34.501374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:34.501346 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:51:36.037234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:36.037175 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:36.045226 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:36.045197 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp"] Oct 31 00:51:36.058155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:36.058126 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c128725-v7nzp"] Oct 31 00:51:36.218571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:36.218529 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:36.428042 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:36.428007 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49c87f5-dcd1-4c71-824e-2d4d70290883" path="/var/lib/kubelet/pods/b49c87f5-dcd1-4c71-824e-2d4d70290883/volumes" Oct 31 00:51:36.490300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:36.490255 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:37.896498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:37.896413 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:38.058601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:38.058562 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:38.431718 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:38.431673 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:38.581811 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:38.581776 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:38.789517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:38.789408 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:39.620388 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:39.620350 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:41.766779 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:41.766744 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:43.628951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:43.628912 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:44.964044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:44.963965 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj"] Oct 31 00:51:44.964524 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:44.964204 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" podUID="7603dd32-be15-4bfa-ae0f-c33908131809" containerName="maas-api" containerID="cri-o://150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca" gracePeriod=30 Oct 31 00:51:45.077314 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.077277 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:45.395831 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.395354 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:51:45.491064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.491030 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-metrics-certs\") pod \"7603dd32-be15-4bfa-ae0f-c33908131809\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " Oct 31 00:51:45.491274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.491160 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7603dd32-be15-4bfa-ae0f-c33908131809-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7603dd32-be15-4bfa-ae0f-c33908131809-openshift-service-ca\") pod \"7603dd32-be15-4bfa-ae0f-c33908131809\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " Oct 31 00:51:45.491274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.491208 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-maas-api-tls\") pod \"7603dd32-be15-4bfa-ae0f-c33908131809\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " Oct 31 00:51:45.491646 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.491609 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7603dd32-be15-4bfa-ae0f-c33908131809-openshift-service-ca" pod "7603dd32-be15-4bfa-ae0f-c33908131809" (UID: "7603dd32-be15-4bfa-ae0f-c33908131809"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:45.492023 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.491719 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7603dd32-be15-4bfa-ae0f-c33908131809-kube-api-access-xhwwf\" (UniqueName: \"kubernetes.io/projected/7603dd32-be15-4bfa-ae0f-c33908131809-kube-api-access-xhwwf\") pod \"7603dd32-be15-4bfa-ae0f-c33908131809\" (UID: \"7603dd32-be15-4bfa-ae0f-c33908131809\") " Oct 31 00:51:45.492865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.492842 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7603dd32-be15-4bfa-ae0f-c33908131809-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:45.494180 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.494156 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-metrics-certs" pod "7603dd32-be15-4bfa-ae0f-c33908131809" (UID: "7603dd32-be15-4bfa-ae0f-c33908131809"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:45.494268 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.494182 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-maas-api-tls" pod "7603dd32-be15-4bfa-ae0f-c33908131809" (UID: "7603dd32-be15-4bfa-ae0f-c33908131809"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:45.494268 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.494219 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7603dd32-be15-4bfa-ae0f-c33908131809-kube-api-access-xhwwf" pod "7603dd32-be15-4bfa-ae0f-c33908131809" (UID: "7603dd32-be15-4bfa-ae0f-c33908131809"). InnerVolumeSpecName "kube-api-access-xhwwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:45.594131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.594037 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:45.594131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.594069 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwwf\" (UniqueName: \"kubernetes.io/projected/7603dd32-be15-4bfa-ae0f-c33908131809-kube-api-access-xhwwf\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:45.594131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.594080 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7603dd32-be15-4bfa-ae0f-c33908131809-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:51:45.616918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.616829 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" Oct 31 00:51:45.658688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.658382 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" event={"ID":"7603dd32-be15-4bfa-ae0f-c33908131809","Type":"ContainerDied","Data":"150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca"} Oct 31 00:51:45.658688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.658483 5978 scope.go:122] "RemoveContainer" containerID="150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca" Oct 31 00:51:45.684614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.616714 5978 generic.go:356] "Generic (PLEG): container finished" podID="7603dd32-be15-4bfa-ae0f-c33908131809" containerID="150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca" exitCode=0 Oct 31 00:51:45.684614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.678092 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj" event={"ID":"7603dd32-be15-4bfa-ae0f-c33908131809","Type":"ContainerDied","Data":"08f5b574f3f742915f0f44d2858df901e1f3873b611fed16b7848f384e4b1907"} Oct 31 00:51:45.692699 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.692665 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:45.758985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.758546 5978 scope.go:122] "RemoveContainer" containerID="150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca" Oct 31 00:51:45.760155 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:51:45.759583 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca\": container with ID starting with 150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca not found: ID does not exist" containerID="150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca" Oct 31 00:51:45.760155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.759623 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca"} err="failed to get container status \"150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca\": rpc error: code = NotFound desc = could not find container \"150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca\": container with ID starting with 150634560b64cc54fe2075a812155271c9c35c817c32d34e713f513875a8c6ca not found: ID does not exist" Oct 31 00:51:45.762923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.762862 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj"] Oct 31 00:51:45.769494 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.768213 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e166ca-65cb8fb655-6jtjj"] Oct 31 00:51:45.890492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:45.890388 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:46.033962 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:46.033909 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:46.182352 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:46.182261 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:46.427248 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:46.427209 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7603dd32-be15-4bfa-ae0f-c33908131809" path="/var/lib/kubelet/pods/7603dd32-be15-4bfa-ae0f-c33908131809/volumes" Oct 31 00:51:46.471542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:46.471411 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:47.414406 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:47.414363 5978 scope.go:122] "RemoveContainer" containerID="02c4267aed3b09e1221019502f5f106d26ca750a29720718b64313872ba13e03" Oct 31 00:51:47.429143 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:47.429117 5978 scope.go:122] "RemoveContainer" containerID="b29a7126773042fc537ba17401eb881fa48adefd2c1cd7e659611386ebd5a002" Oct 31 00:51:47.440690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:47.440670 5978 scope.go:122] "RemoveContainer" containerID="3b038947aa52e5f8b4cf445ed2e498d6c567deec5d27a828ebf57d9f0e586ca6" Oct 31 00:51:48.227518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:48.227435 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:49.946339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:49.946291 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:50.102189 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:50.102148 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:51.891664 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:51.891625 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:53.465506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.465469 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4"] Oct 31 00:51:53.466502 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.466480 5978 cpu_manager.go:414] "containerMap: removing container" podUID="7603dd32-be15-4bfa-ae0f-c33908131809" containerName="maas-api" Oct 31 00:51:53.466502 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.466502 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7603dd32-be15-4bfa-ae0f-c33908131809" containerName="maas-api" Oct 31 00:51:53.466764 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.466747 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="7603dd32-be15-4bfa-ae0f-c33908131809" containerName="maas-api" Oct 31 00:51:53.470899 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.470878 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.493855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.493814 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4"] Oct 31 00:51:53.498729 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.498700 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4d5621-serving-cert\"" Oct 31 00:51:53.531750 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.531712 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:53.660133 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.660099 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.660343 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.660256 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.660416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.660357 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-metrics-certs\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.660498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.660478 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxkwz\" (UniqueName: \"kubernetes.io/projected/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-kube-api-access-cxkwz\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.683659 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.683630 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:53.770650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.766095 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.770650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.766340 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.770650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.766410 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-metrics-certs\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.770650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.766495 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxkwz\" (UniqueName: \"kubernetes.io/projected/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-kube-api-access-cxkwz\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.773569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.773540 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.792495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.775984 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.792495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.776102 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-metrics-certs\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.802835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.799573 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxkwz\" (UniqueName: \"kubernetes.io/projected/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-kube-api-access-cxkwz\") pod \"maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:53.855010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:53.854975 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:54.062176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:54.062077 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:54.084220 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:54.084188 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:54.414144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:54.414111 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4"] Oct 31 00:51:54.414299 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:51:54.414275 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639a34f2_fa2a_42c6_9c2f_cdf2664495ee.slice/crio-e430d2ef7028d3f72aa325dec697ea534dc147c65f8aeae718ab95224d23b8d2 WatchSource:0}: Error finding container e430d2ef7028d3f72aa325dec697ea534dc147c65f8aeae718ab95224d23b8d2: Status 404 returned error can't find the container with id e430d2ef7028d3f72aa325dec697ea534dc147c65f8aeae718ab95224d23b8d2 Oct 31 00:51:54.415856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:54.415837 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:51:54.857685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:54.857648 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" event={"ID":"639a34f2-fa2a-42c6-9c2f-cdf2664495ee","Type":"ContainerStarted","Data":"e430d2ef7028d3f72aa325dec697ea534dc147c65f8aeae718ab95224d23b8d2"} Oct 31 00:51:55.815781 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:55.815738 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:55.868951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:55.868906 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" event={"ID":"639a34f2-fa2a-42c6-9c2f-cdf2664495ee","Type":"ContainerStarted","Data":"8aa4045ea75387fc50dbacf296cf9bcf4f83152bf753308429a681856f709311"} Oct 31 00:51:55.869351 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:55.868985 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:51:55.908127 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:55.908078 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" podStartSLOduration=2.443952709 podStartE2EDuration="2.908063273s" totalImagesPullingTime="464.110564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:55.907100088 +0000 UTC m=+1933.605213852" watchObservedRunningTime="2026-10-31 00:51:55.908063273 +0000 UTC m=+1933.606177038" Oct 31 00:51:57.605348 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:57.605304 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:57.760442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:57.760406 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:57.904835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:57.904746 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:59.686605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:51:59.686568 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:00.560838 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:00.560805 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:00.699494 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:00.699448 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:01.749264 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:01.749226 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:01.884339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:01.884309 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 00:52:03.182866 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:52:03.182824 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:52:03.380206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:03.380161 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:04.181465 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:04.181416 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth"] Oct 31 00:52:04.181751 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:04.181725 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" containerID="cri-o://3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125" gracePeriod=30 Oct 31 00:52:05.049384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:05.049346 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn"] Oct 31 00:52:05.049919 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:05.049580 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" containerID="cri-o://86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6" gracePeriod=30 Oct 31 00:52:05.166820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:05.166762 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:05.480237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:05.480201 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw"] Oct 31 00:52:05.480431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:05.480415 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" containerID="cri-o://ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891" gracePeriod=30 Oct 31 00:52:05.495596 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:05.495554 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:07.103393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:07.103351 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:07.267651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:07.267605 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:08.884614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:08.884573 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:09.501931 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.501898 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:52:09.599249 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599204 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-credential-socket\") pod \"a8b37e6d-25c3-4b3d-975b-496139791738\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " Oct 31 00:52:09.599496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599313 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8b37e6d-25c3-4b3d-975b-496139791738-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b37e6d-25c3-4b3d-975b-496139791738-istio-podinfo\") pod \"a8b37e6d-25c3-4b3d-975b-496139791738\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " Oct 31 00:52:09.599496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599368 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-certs\") pod \"a8b37e6d-25c3-4b3d-975b-496139791738\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " Oct 31 00:52:09.599496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599401 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-kube-api-access-s26hs\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-kube-api-access-s26hs\") pod \"a8b37e6d-25c3-4b3d-975b-496139791738\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " Oct 31 00:52:09.599698 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599538 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-socket\") pod \"a8b37e6d-25c3-4b3d-975b-496139791738\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " Oct 31 00:52:09.599698 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599618 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-data\") pod \"a8b37e6d-25c3-4b3d-975b-496139791738\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " Oct 31 00:52:09.599698 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599636 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-credential-socket" pod "a8b37e6d-25c3-4b3d-975b-496139791738" (UID: "a8b37e6d-25c3-4b3d-975b-496139791738"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.599864 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599761 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b37e6d-25c3-4b3d-975b-496139791738-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b37e6d-25c3-4b3d-975b-496139791738-istiod-ca-cert\") pod \"a8b37e6d-25c3-4b3d-975b-496139791738\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " Oct 31 00:52:09.599864 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599824 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-istio-token\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-istio-token\") pod \"a8b37e6d-25c3-4b3d-975b-496139791738\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " Oct 31 00:52:09.599987 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.599878 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-envoy\") pod \"a8b37e6d-25c3-4b3d-975b-496139791738\" (UID: \"a8b37e6d-25c3-4b3d-975b-496139791738\") " Oct 31 00:52:09.600327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.600301 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-data" pod "a8b37e6d-25c3-4b3d-975b-496139791738" (UID: "a8b37e6d-25c3-4b3d-975b-496139791738"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.600690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.600667 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-socket" pod "a8b37e6d-25c3-4b3d-975b-496139791738" (UID: "a8b37e6d-25c3-4b3d-975b-496139791738"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.600876 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.600851 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.600956 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.600881 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.601293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.601271 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-certs" pod "a8b37e6d-25c3-4b3d-975b-496139791738" (UID: "a8b37e6d-25c3-4b3d-975b-496139791738"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.601403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.601291 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b37e6d-25c3-4b3d-975b-496139791738-istiod-ca-cert" pod "a8b37e6d-25c3-4b3d-975b-496139791738" (UID: "a8b37e6d-25c3-4b3d-975b-496139791738"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:09.604305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.602411 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-kube-api-access-s26hs" pod "a8b37e6d-25c3-4b3d-975b-496139791738" (UID: "a8b37e6d-25c3-4b3d-975b-496139791738"). InnerVolumeSpecName "kube-api-access-s26hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:09.606383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.606342 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8b37e6d-25c3-4b3d-975b-496139791738-istio-podinfo" pod "a8b37e6d-25c3-4b3d-975b-496139791738" (UID: "a8b37e6d-25c3-4b3d-975b-496139791738"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:09.606847 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.606812 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-istio-token" pod "a8b37e6d-25c3-4b3d-975b-496139791738" (UID: "a8b37e6d-25c3-4b3d-975b-496139791738"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:09.617087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.617042 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-envoy" pod "a8b37e6d-25c3-4b3d-975b-496139791738" (UID: "a8b37e6d-25c3-4b3d-975b-496139791738"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:09.702966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.702930 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b37e6d-25c3-4b3d-975b-496139791738-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.702966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.702959 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.702966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.702968 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.702966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.702976 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b37e6d-25c3-4b3d-975b-496139791738-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.703293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.702984 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.703293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.702994 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s26hs\" (UniqueName: \"kubernetes.io/projected/a8b37e6d-25c3-4b3d-975b-496139791738-kube-api-access-s26hs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:09.703293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:09.703003 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b37e6d-25c3-4b3d-975b-496139791738-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.124122 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.124083 5978 generic.go:356] "Generic (PLEG): container finished" podID="a8b37e6d-25c3-4b3d-975b-496139791738" containerID="3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125" exitCode=0 Oct 31 00:52:10.124685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.124143 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" Oct 31 00:52:10.124685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.124167 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" event={"ID":"a8b37e6d-25c3-4b3d-975b-496139791738","Type":"ContainerDied","Data":"3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125"} Oct 31 00:52:10.124685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.124239 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth" event={"ID":"a8b37e6d-25c3-4b3d-975b-496139791738","Type":"ContainerDied","Data":"90a14bf49b78e210cb0b1dd1a54a15fc09b69b30f7090cb32e41f1c67c6a2710"} Oct 31 00:52:10.124685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.124266 5978 scope.go:122] "RemoveContainer" containerID="3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125" Oct 31 00:52:10.176028 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.176001 5978 scope.go:122] "RemoveContainer" containerID="3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125" Oct 31 00:52:10.176602 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:52:10.176440 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125\": container with ID starting with 3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125 not found: ID does not exist" containerID="3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125" Oct 31 00:52:10.176602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.176575 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125"} err="failed to get container status \"3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125\": rpc error: code = NotFound desc = could not find container \"3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125\": container with ID starting with 3fcdac82c9db0a5828c366134e49085d012e5912a8c5edc95ec511b696931125 not found: ID does not exist" Oct 31 00:52:10.183313 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.183282 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth"] Oct 31 00:52:10.189728 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.189696 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7whth"] Oct 31 00:52:10.382318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.382286 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:52:10.437351 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.437319 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" path="/var/lib/kubelet/pods/a8b37e6d-25c3-4b3d-975b-496139791738/volumes" Oct 31 00:52:10.518231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.518198 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-kube-api-access-j4f7c\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-kube-api-access-j4f7c\") pod \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " Oct 31 00:52:10.518368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.518314 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-data\") pod \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " Oct 31 00:52:10.518432 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.518406 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-envoy\") pod \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " Oct 31 00:52:10.518704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.518634 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-credential-socket\") pod \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " Oct 31 00:52:10.518704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.518681 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-token\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-token\") pod \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " Oct 31 00:52:10.518906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.518887 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-podinfo\") pod \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " Oct 31 00:52:10.518987 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.518972 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-socket\") pod \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " Oct 31 00:52:10.519074 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.519057 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-certs\") pod \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " Oct 31 00:52:10.519134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.519108 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istiod-ca-cert\") pod \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\" (UID: \"1a594534-8d06-484c-8cd1-17a4a9fe3bd2\") " Oct 31 00:52:10.519297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.519268 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-credential-socket" pod "1a594534-8d06-484c-8cd1-17a4a9fe3bd2" (UID: "1a594534-8d06-484c-8cd1-17a4a9fe3bd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.523044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.519462 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-data" pod "1a594534-8d06-484c-8cd1-17a4a9fe3bd2" (UID: "1a594534-8d06-484c-8cd1-17a4a9fe3bd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.523044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.519700 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-socket" pod "1a594534-8d06-484c-8cd1-17a4a9fe3bd2" (UID: "1a594534-8d06-484c-8cd1-17a4a9fe3bd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.523044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.519791 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istiod-ca-cert" pod "1a594534-8d06-484c-8cd1-17a4a9fe3bd2" (UID: "1a594534-8d06-484c-8cd1-17a4a9fe3bd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:10.523044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.520052 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-certs" pod "1a594534-8d06-484c-8cd1-17a4a9fe3bd2" (UID: "1a594534-8d06-484c-8cd1-17a4a9fe3bd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.523044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.520630 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.523044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.520653 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.523044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.520668 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.523044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.520680 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.523044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.520693 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.526833 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.526800 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-envoy" pod "1a594534-8d06-484c-8cd1-17a4a9fe3bd2" (UID: "1a594534-8d06-484c-8cd1-17a4a9fe3bd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.527418 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.527392 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-kube-api-access-j4f7c" pod "1a594534-8d06-484c-8cd1-17a4a9fe3bd2" (UID: "1a594534-8d06-484c-8cd1-17a4a9fe3bd2"). InnerVolumeSpecName "kube-api-access-j4f7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:10.529906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.529874 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-podinfo" pod "1a594534-8d06-484c-8cd1-17a4a9fe3bd2" (UID: "1a594534-8d06-484c-8cd1-17a4a9fe3bd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:10.532802 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.532772 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-token" pod "1a594534-8d06-484c-8cd1-17a4a9fe3bd2" (UID: "1a594534-8d06-484c-8cd1-17a4a9fe3bd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:10.551072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.544167 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:10.624582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.624526 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.624582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.624566 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.624582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.624580 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.625084 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.624594 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4f7c\" (UniqueName: \"kubernetes.io/projected/1a594534-8d06-484c-8cd1-17a4a9fe3bd2-kube-api-access-j4f7c\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:10.798405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.798376 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:52:10.929930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.929809 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-podinfo\") pod \"a7ed5b38-29a4-42e8-933f-c7b305641872\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " Oct 31 00:52:10.930109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.929948 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-certs\") pod \"a7ed5b38-29a4-42e8-933f-c7b305641872\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " Oct 31 00:52:10.930109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.929997 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ed5b38-29a4-42e8-933f-c7b305641872-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ed5b38-29a4-42e8-933f-c7b305641872-istiod-ca-cert\") pod \"a7ed5b38-29a4-42e8-933f-c7b305641872\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " Oct 31 00:52:10.930203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.930109 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-data\") pod \"a7ed5b38-29a4-42e8-933f-c7b305641872\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " Oct 31 00:52:10.930282 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.930265 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-credential-socket\") pod \"a7ed5b38-29a4-42e8-933f-c7b305641872\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " Oct 31 00:52:10.930324 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.930314 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-envoy\") pod \"a7ed5b38-29a4-42e8-933f-c7b305641872\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " Oct 31 00:52:10.930367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.930356 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-token\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-token\") pod \"a7ed5b38-29a4-42e8-933f-c7b305641872\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " Oct 31 00:52:10.930424 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.930408 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-kube-api-access-7pwkj\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-kube-api-access-7pwkj\") pod \"a7ed5b38-29a4-42e8-933f-c7b305641872\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " Oct 31 00:52:10.930586 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.930569 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-socket\") pod \"a7ed5b38-29a4-42e8-933f-c7b305641872\" (UID: \"a7ed5b38-29a4-42e8-933f-c7b305641872\") " Oct 31 00:52:10.934824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.934786 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-credential-socket" pod "a7ed5b38-29a4-42e8-933f-c7b305641872" (UID: "a7ed5b38-29a4-42e8-933f-c7b305641872"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.935325 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.935299 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-data" pod "a7ed5b38-29a4-42e8-933f-c7b305641872" (UID: "a7ed5b38-29a4-42e8-933f-c7b305641872"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.935430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.935368 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-socket" pod "a7ed5b38-29a4-42e8-933f-c7b305641872" (UID: "a7ed5b38-29a4-42e8-933f-c7b305641872"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.936186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.936162 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-certs" pod "a7ed5b38-29a4-42e8-933f-c7b305641872" (UID: "a7ed5b38-29a4-42e8-933f-c7b305641872"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.937533 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.937511 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-token" pod "a7ed5b38-29a4-42e8-933f-c7b305641872" (UID: "a7ed5b38-29a4-42e8-933f-c7b305641872"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:10.938371 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.938348 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-podinfo" pod "a7ed5b38-29a4-42e8-933f-c7b305641872" (UID: "a7ed5b38-29a4-42e8-933f-c7b305641872"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:10.938860 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.938837 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-envoy" pod "a7ed5b38-29a4-42e8-933f-c7b305641872" (UID: "a7ed5b38-29a4-42e8-933f-c7b305641872"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:10.939314 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.939298 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-kube-api-access-7pwkj" pod "a7ed5b38-29a4-42e8-933f-c7b305641872" (UID: "a7ed5b38-29a4-42e8-933f-c7b305641872"). InnerVolumeSpecName "kube-api-access-7pwkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:10.940081 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:10.940055 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ed5b38-29a4-42e8-933f-c7b305641872-istiod-ca-cert" pod "a7ed5b38-29a4-42e8-933f-c7b305641872" (UID: "a7ed5b38-29a4-42e8-933f-c7b305641872"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:11.033566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.033419 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:11.033566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.033469 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:11.033566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.033485 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:11.033566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.033499 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:11.033566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.033515 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwkj\" (UniqueName: \"kubernetes.io/projected/a7ed5b38-29a4-42e8-933f-c7b305641872-kube-api-access-7pwkj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:11.033566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.033529 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:11.033566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.033542 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ed5b38-29a4-42e8-933f-c7b305641872-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:11.033566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.033555 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ed5b38-29a4-42e8-933f-c7b305641872-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:11.033566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.033566 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ed5b38-29a4-42e8-933f-c7b305641872-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:52:11.133290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.133256 5978 generic.go:356] "Generic (PLEG): container finished" podID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerID="86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6" exitCode=0 Oct 31 00:52:11.133290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.133300 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" Oct 31 00:52:11.133877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.133340 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" event={"ID":"1a594534-8d06-484c-8cd1-17a4a9fe3bd2","Type":"ContainerDied","Data":"86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6"} Oct 31 00:52:11.133877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.133403 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn" event={"ID":"1a594534-8d06-484c-8cd1-17a4a9fe3bd2","Type":"ContainerDied","Data":"51c9f12a2f757f7858598d7be25fac067f0fbef2447b455d8b3e5363736e86c0"} Oct 31 00:52:11.133877 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.133441 5978 scope.go:122] "RemoveContainer" containerID="86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6" Oct 31 00:52:11.136856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.136825 5978 generic.go:356] "Generic (PLEG): container finished" podID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerID="ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891" exitCode=0 Oct 31 00:52:11.136980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.136878 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" Oct 31 00:52:11.136980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.136901 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" event={"ID":"a7ed5b38-29a4-42e8-933f-c7b305641872","Type":"ContainerDied","Data":"ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891"} Oct 31 00:52:11.136980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.136948 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw" event={"ID":"a7ed5b38-29a4-42e8-933f-c7b305641872","Type":"ContainerDied","Data":"0919e2872bb2c696968f180441f9bdcba3d53f1acc82a2dd5bf4cd7378c2f5ff"} Oct 31 00:52:11.146072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.146040 5978 scope.go:122] "RemoveContainer" containerID="86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6" Oct 31 00:52:11.146567 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:52:11.146417 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6\": container with ID starting with 86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6 not found: ID does not exist" containerID="86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6" Oct 31 00:52:11.146567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.146443 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6"} err="failed to get container status \"86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6\": rpc error: code = NotFound desc = could not find container \"86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6\": container with ID starting with 86b22db9779bd2c927f761876813306711f1a9422595f9b70794239de259c9d6 not found: ID does not exist" Oct 31 00:52:11.146567 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.146505 5978 scope.go:122] "RemoveContainer" containerID="ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891" Oct 31 00:52:11.158652 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.158628 5978 scope.go:122] "RemoveContainer" containerID="750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4" Oct 31 00:52:11.173463 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.173430 5978 scope.go:122] "RemoveContainer" containerID="ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891" Oct 31 00:52:11.173835 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:52:11.173813 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891\": container with ID starting with ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891 not found: ID does not exist" containerID="ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891" Oct 31 00:52:11.173880 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.173847 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891"} err="failed to get container status \"ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891\": rpc error: code = NotFound desc = could not find container \"ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891\": container with ID starting with ce35d8e2bfab09b94bdf82985dfa78d7ca1fb5c9a705ccb9bcba24f682989891 not found: ID does not exist" Oct 31 00:52:11.173880 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.173875 5978 scope.go:122] "RemoveContainer" containerID="750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4" Oct 31 00:52:11.174157 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:52:11.174133 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4\": container with ID starting with 750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4 not found: ID does not exist" containerID="750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4" Oct 31 00:52:11.174197 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.174166 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4"} err="failed to get container status \"750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4\": rpc error: code = NotFound desc = could not find container \"750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4\": container with ID starting with 750bb4a478b4dd1577e17606c840a1fc32ee37033dcccb6b9f739c1d6a7d8ce4 not found: ID does not exist" Oct 31 00:52:11.193847 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.193751 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn"] Oct 31 00:52:11.197830 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.197800 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-kqlkn"] Oct 31 00:52:11.226668 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.226631 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw"] Oct 31 00:52:11.231095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:11.231064 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-48jkw"] Oct 31 00:52:12.195843 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:12.195808 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:12.427563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:12.427528 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" path="/var/lib/kubelet/pods/1a594534-8d06-484c-8cd1-17a4a9fe3bd2/volumes" Oct 31 00:52:12.428662 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:12.428644 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" path="/var/lib/kubelet/pods/a7ed5b38-29a4-42e8-933f-c7b305641872/volumes" Oct 31 00:52:13.689811 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:13.689772 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:14.385974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:14.385942 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:14.574231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:14.574114 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:14.748383 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:14.748279 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:15.145045 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:15.145005 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:16.721534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:16.721486 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:16.908176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:16.908121 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:18.835281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:18.835236 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:20.081018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:20.080968 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:20.588124 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:20.588086 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:22.118666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:22.118625 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:23.901799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:23.901747 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:25.492956 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:25.492919 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:27.178754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:27.176702 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:29.362732 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:29.362686 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:31.311356 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:31.311314 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:33.184610 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:52:33.184573 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:52:33.472116 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:33.472021 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:35.687019 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:35.686978 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:37.101092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:37.101049 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:37.563197 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:37.563101 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:37.704868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:37.704825 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:38.086944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:38.086898 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:38.305375 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:38.305335 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:38.456565 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:38.456472 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:38.873184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:38.873148 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:39.252535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.252409 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq"] Oct 31 00:52:39.253331 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253308 5978 cpu_manager.go:414] "containerMap: removing container" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" Oct 31 00:52:39.253331 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253328 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" Oct 31 00:52:39.253581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253384 5978 cpu_manager.go:414] "containerMap: removing container" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" Oct 31 00:52:39.253581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253392 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" Oct 31 00:52:39.253581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253430 5978 cpu_manager.go:414] "containerMap: removing container" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" Oct 31 00:52:39.253581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253437 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" Oct 31 00:52:39.253581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253487 5978 cpu_manager.go:414] "containerMap: removing container" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" Oct 31 00:52:39.253581 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253494 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" Oct 31 00:52:39.253994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253715 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b37e6d-25c3-4b3d-975b-496139791738" containerName="istio-proxy" Oct 31 00:52:39.253994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253734 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" Oct 31 00:52:39.253994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253740 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ed5b38-29a4-42e8-933f-c7b305641872" containerName="istio-proxy" Oct 31 00:52:39.253994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.253750 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a594534-8d06-484c-8cd1-17a4a9fe3bd2" containerName="istio-proxy" Oct 31 00:52:39.258197 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.258168 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.287522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.287487 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq"] Oct 31 00:52:39.291749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.291722 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bdb61f-openshift-default-dockercfg-x5q5s\"" Oct 31 00:52:39.340848 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.340810 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:39.397179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.397137 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f3833c-293b-41e4-878a-a36a42207ff2-istio-podinfo\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.397403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.397211 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwmnn\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-kube-api-access-wwmnn\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.397403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.397264 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-istio-token\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.397403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.397339 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-credential-socket\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.397403 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.397383 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-envoy\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.397666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.397571 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-data\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.397666 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.397653 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-socket\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.397754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.397725 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f3833c-293b-41e4-878a-a36a42207ff2-istiod-ca-cert\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.397850 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.397832 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-certs\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.504937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.499151 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-data\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.504937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.499232 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-socket\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.504937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.499301 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f3833c-293b-41e4-878a-a36a42207ff2-istiod-ca-cert\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.504937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.499402 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-certs\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.504937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.499534 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f3833c-293b-41e4-878a-a36a42207ff2-istio-podinfo\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.504937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.499574 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwmnn\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-kube-api-access-wwmnn\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.504937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.499627 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-istio-token\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.504937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.499732 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-credential-socket\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.504937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.499776 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-envoy\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.538615 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.523793 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-socket\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.538615 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.525029 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-data\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.538615 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.525242 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-certs\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.538615 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.525845 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-credential-socket\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.538615 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.527387 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-envoy\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.548357 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.542819 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f3833c-293b-41e4-878a-a36a42207ff2-istiod-ca-cert\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.558263 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.558193 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-istio-token\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.561686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.560963 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwmnn\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-kube-api-access-wwmnn\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.580485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.562941 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:39.589156 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.589058 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f3833c-293b-41e4-878a-a36a42207ff2-istio-podinfo\") pod \"e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:39.876308 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:39.876265 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:40.037478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:40.036796 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:40.204214 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:40.204104 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq"] Oct 31 00:52:40.220489 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:52:40.207563 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f3833c_293b_41e4_878a_a36a42207ff2.slice/crio-dd533d6f308162f01d0482ceb558c0576406103191b4c52f8c0d015164cf9c4f WatchSource:0}: Error finding container dd533d6f308162f01d0482ceb558c0576406103191b4c52f8c0d015164cf9c4f: Status 404 returned error can't find the container with id dd533d6f308162f01d0482ceb558c0576406103191b4c52f8c0d015164cf9c4f Oct 31 00:52:40.220489 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:40.214718 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:52:40.220489 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:40.214832 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:52:40.220489 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:40.214887 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:52:40.562620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:40.562520 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" event={"ID":"36f3833c-293b-41e4-878a-a36a42207ff2","Type":"ContainerStarted","Data":"9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823"} Oct 31 00:52:40.562620 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:40.562587 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" event={"ID":"36f3833c-293b-41e4-878a-a36a42207ff2","Type":"ContainerStarted","Data":"dd533d6f308162f01d0482ceb558c0576406103191b4c52f8c0d015164cf9c4f"} Oct 31 00:52:40.605291 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:40.605218 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" podStartSLOduration=1.605198858 podStartE2EDuration="1.605198858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:40.604188319 +0000 UTC m=+1978.302302108" watchObservedRunningTime="2026-10-31 00:52:40.605198858 +0000 UTC m=+1978.303312623" Oct 31 00:52:40.876447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:40.876411 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:41.883777 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:41.883743 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:41.919112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:41.919062 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:42.563362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:42.563327 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:42.579831 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:42.579798 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:42.580825 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:42.580799 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:52:42.712076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:42.712040 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:43.164749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:43.164716 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:43.323190 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:43.323154 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:44.220524 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:44.218569 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:44.702471 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:44.702415 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:44.846240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:44.846198 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:46.309718 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:46.309684 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:46.477059 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:46.476980 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:47.524980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:47.524942 5978 scope.go:122] "RemoveContainer" containerID="ca9fdb304360757c8e045ef719a44db3f72f326a1ee294447e3b28e0fb669b46" Oct 31 00:52:47.559655 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:47.559620 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:48.036114 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.036069 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:48.476348 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.476292 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:48.663538 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.663496 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g"] Oct 31 00:52:48.669077 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.669016 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:48.674283 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.674245 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:52:48.694617 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.694574 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g"] Oct 31 00:52:48.698531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.697673 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bdb61f\"" Oct 31 00:52:48.717530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.717499 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bdb61f-dockercfg-zggtj\"" Oct 31 00:52:48.783540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.782950 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44gnp\" (UniqueName: \"kubernetes.io/projected/47678034-2289-4238-8e31-074e2c1e5c78-kube-api-access-44gnp\") pod \"payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g\" (UID: \"47678034-2289-4238-8e31-074e2c1e5c78\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:52:48.783540 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.783249 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47678034-2289-4238-8e31-074e2c1e5c78-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g\" (UID: \"47678034-2289-4238-8e31-074e2c1e5c78\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:52:48.831446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.825936 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:48.885759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.885720 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44gnp\" (UniqueName: \"kubernetes.io/projected/47678034-2289-4238-8e31-074e2c1e5c78-kube-api-access-44gnp\") pod \"payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g\" (UID: \"47678034-2289-4238-8e31-074e2c1e5c78\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:52:48.886665 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.886629 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47678034-2289-4238-8e31-074e2c1e5c78-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g\" (UID: \"47678034-2289-4238-8e31-074e2c1e5c78\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:52:48.894954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.887775 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47678034-2289-4238-8e31-074e2c1e5c78-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g\" (UID: \"47678034-2289-4238-8e31-074e2c1e5c78\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:52:48.912303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:48.912268 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44gnp\" (UniqueName: \"kubernetes.io/projected/47678034-2289-4238-8e31-074e2c1e5c78-kube-api-access-44gnp\") pod \"payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g\" (UID: \"47678034-2289-4238-8e31-074e2c1e5c78\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:52:49.002186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:49.002155 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:52:49.150230 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:49.150191 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:49.349955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:49.349917 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g"] Oct 31 00:52:49.350797 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:52:49.350765 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47678034_2289_4238_8e31_074e2c1e5c78.slice/crio-5f4bfae22f3861b16ca57630d1e4df57784ab4b103bbf2f227596ccb758a6143 WatchSource:0}: Error finding container 5f4bfae22f3861b16ca57630d1e4df57784ab4b103bbf2f227596ccb758a6143: Status 404 returned error can't find the container with id 5f4bfae22f3861b16ca57630d1e4df57784ab4b103bbf2f227596ccb758a6143 Oct 31 00:52:49.640173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:49.640074 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" event={"ID":"47678034-2289-4238-8e31-074e2c1e5c78","Type":"ContainerStarted","Data":"1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf"} Oct 31 00:52:49.640173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:49.640143 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" event={"ID":"47678034-2289-4238-8e31-074e2c1e5c78","Type":"ContainerStarted","Data":"5f4bfae22f3861b16ca57630d1e4df57784ab4b103bbf2f227596ccb758a6143"} Oct 31 00:52:49.640173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:49.640172 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:52:49.681329 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:49.681270 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" podStartSLOduration=1.681249812 podStartE2EDuration="1.681249812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:49.679361147 +0000 UTC m=+1987.377474911" watchObservedRunningTime="2026-10-31 00:52:49.681249812 +0000 UTC m=+1987.379363600" Oct 31 00:52:51.345478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:51.341851 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:53.015064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:53.015024 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:53.210609 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:53.210555 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:53.360951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:53.359426 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:54.824679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:54.824645 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:54.975927 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:54.975860 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:55.342698 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:55.342660 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:56.141224 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:56.141183 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:57.645307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:57.645263 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:57.963225 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:57.963131 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:58.128858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:58.128813 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:59.492311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:52:59.492271 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:00.661981 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:00.661950 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:53:01.431606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:01.431569 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:02.986671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:02.986619 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:03.185552 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:53:03.185513 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:53:04.585704 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:04.585666 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:04.736485 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:04.736426 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:04.917724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:04.917620 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97"] Oct 31 00:53:04.917935 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:04.917906 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" containerID="cri-o://dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385" gracePeriod=30 Oct 31 00:53:05.063478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:05.048595 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:06.479865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:06.479827 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:08.011471 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:08.011419 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:08.898476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:08.897844 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:09.129478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:09.127918 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:09.284165 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:09.284058 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:09.640297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:09.640261 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:10.281163 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.281102 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:53:10.342159 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342126 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-data\") pod \"9171fc04-6c22-4ca2-a581-bbff5c77205a\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " Oct 31 00:53:10.342349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342193 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-certs\") pod \"9171fc04-6c22-4ca2-a581-bbff5c77205a\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " Oct 31 00:53:10.342349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342224 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9171fc04-6c22-4ca2-a581-bbff5c77205a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9171fc04-6c22-4ca2-a581-bbff5c77205a-istiod-ca-cert\") pod \"9171fc04-6c22-4ca2-a581-bbff5c77205a\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " Oct 31 00:53:10.342582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342367 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-envoy\") pod \"9171fc04-6c22-4ca2-a581-bbff5c77205a\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " Oct 31 00:53:10.342582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342410 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-token\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-token\") pod \"9171fc04-6c22-4ca2-a581-bbff5c77205a\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " Oct 31 00:53:10.342582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342504 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-kube-api-access-gkl26\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-kube-api-access-gkl26\") pod \"9171fc04-6c22-4ca2-a581-bbff5c77205a\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " Oct 31 00:53:10.342582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342559 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-podinfo\") pod \"9171fc04-6c22-4ca2-a581-bbff5c77205a\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " Oct 31 00:53:10.342799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342593 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-credential-socket\") pod \"9171fc04-6c22-4ca2-a581-bbff5c77205a\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " Oct 31 00:53:10.342799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342646 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-socket\") pod \"9171fc04-6c22-4ca2-a581-bbff5c77205a\" (UID: \"9171fc04-6c22-4ca2-a581-bbff5c77205a\") " Oct 31 00:53:10.342799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342655 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-certs" pod "9171fc04-6c22-4ca2-a581-bbff5c77205a" (UID: "9171fc04-6c22-4ca2-a581-bbff5c77205a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:10.342799 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.342755 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9171fc04-6c22-4ca2-a581-bbff5c77205a-istiod-ca-cert" pod "9171fc04-6c22-4ca2-a581-bbff5c77205a" (UID: "9171fc04-6c22-4ca2-a581-bbff5c77205a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:10.343210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.343185 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-socket" pod "9171fc04-6c22-4ca2-a581-bbff5c77205a" (UID: "9171fc04-6c22-4ca2-a581-bbff5c77205a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:10.343338 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.343313 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-credential-socket" pod "9171fc04-6c22-4ca2-a581-bbff5c77205a" (UID: "9171fc04-6c22-4ca2-a581-bbff5c77205a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:10.343653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.343627 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-data" pod "9171fc04-6c22-4ca2-a581-bbff5c77205a" (UID: "9171fc04-6c22-4ca2-a581-bbff5c77205a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:10.344292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.344267 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:10.344405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.344297 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:10.344405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.344311 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9171fc04-6c22-4ca2-a581-bbff5c77205a-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:10.344405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.344325 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:10.344405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.344339 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:10.346243 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.346115 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-podinfo" pod "9171fc04-6c22-4ca2-a581-bbff5c77205a" (UID: "9171fc04-6c22-4ca2-a581-bbff5c77205a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:53:10.346964 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.346940 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-envoy" pod "9171fc04-6c22-4ca2-a581-bbff5c77205a" (UID: "9171fc04-6c22-4ca2-a581-bbff5c77205a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:10.348967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.348936 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-kube-api-access-gkl26" pod "9171fc04-6c22-4ca2-a581-bbff5c77205a" (UID: "9171fc04-6c22-4ca2-a581-bbff5c77205a"). InnerVolumeSpecName "kube-api-access-gkl26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:10.349820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.349792 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-token" pod "9171fc04-6c22-4ca2-a581-bbff5c77205a" (UID: "9171fc04-6c22-4ca2-a581-bbff5c77205a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:10.445554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.445521 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:10.445554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.445552 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:10.445782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.445565 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkl26\" (UniqueName: \"kubernetes.io/projected/9171fc04-6c22-4ca2-a581-bbff5c77205a-kube-api-access-gkl26\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:10.445782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.445578 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9171fc04-6c22-4ca2-a581-bbff5c77205a-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:10.844076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.844039 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:10.966884 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.966211 5978 generic.go:356] "Generic (PLEG): container finished" podID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerID="dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385" exitCode=0 Oct 31 00:53:10.966884 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.966259 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" Oct 31 00:53:10.966884 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.966272 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" event={"ID":"9171fc04-6c22-4ca2-a581-bbff5c77205a","Type":"ContainerDied","Data":"dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385"} Oct 31 00:53:10.966884 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.966393 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97" event={"ID":"9171fc04-6c22-4ca2-a581-bbff5c77205a","Type":"ContainerDied","Data":"57f5995018fa61e94977a3aed9fb3dfce823907a2e08467715429009a5a8ceab"} Oct 31 00:53:10.966884 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.966412 5978 scope.go:122] "RemoveContainer" containerID="dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385" Oct 31 00:53:10.987876 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.987855 5978 scope.go:122] "RemoveContainer" containerID="1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942" Oct 31 00:53:10.999526 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.999508 5978 scope.go:122] "RemoveContainer" containerID="dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385" Oct 31 00:53:10.999821 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:53:10.999801 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385\": container with ID starting with dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385 not found: ID does not exist" containerID="dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385" Oct 31 00:53:10.999890 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.999827 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385"} err="failed to get container status \"dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385\": rpc error: code = NotFound desc = could not find container \"dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385\": container with ID starting with dc758f9bf7ca1c1ffe943d9397f0be2fcbe9ab68a561ddee5563b4cca23f5385 not found: ID does not exist" Oct 31 00:53:10.999890 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:10.999842 5978 scope.go:122] "RemoveContainer" containerID="1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942" Oct 31 00:53:11.000071 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:53:11.000052 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942\": container with ID starting with 1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942 not found: ID does not exist" containerID="1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942" Oct 31 00:53:11.000071 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:11.000070 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942"} err="failed to get container status \"1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942\": rpc error: code = NotFound desc = could not find container \"1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942\": container with ID starting with 1de2bd53701b6da5190f193752c0b49149904b73d5579e64609fecfdbd044942 not found: ID does not exist" Oct 31 00:53:11.042366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:11.042324 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:11.044350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:11.044307 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97"] Oct 31 00:53:11.061143 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:11.061109 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-208e9f32-openshift-default-77466675fc-qqt97"] Oct 31 00:53:11.236497 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:11.236362 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:12.446210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:12.446167 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" path="/var/lib/kubelet/pods/9171fc04-6c22-4ca2-a581-bbff5c77205a/volumes" Oct 31 00:53:12.893811 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:12.893775 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:13.573481 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:13.568760 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:13.736492 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:13.736185 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:14.009856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:14.009766 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:14.368993 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:14.368958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:15.473234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:15.472836 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:16.554444 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:16.554405 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:16.716217 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:16.716176 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:16.917725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:16.917674 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms"] Oct 31 00:53:16.918671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:16.918628 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" containerID="cri-o://13802e888d9ebe99cf0d832460df845156d9f120864a29ebebcc42a376d39990" gracePeriod=30 Oct 31 00:53:17.018199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.018138 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:17.295218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.295130 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:17.431806 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.431756 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m"] Oct 31 00:53:17.432825 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.432801 5978 cpu_manager.go:414] "containerMap: removing container" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" Oct 31 00:53:17.432825 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.432824 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" Oct 31 00:53:17.433017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.432850 5978 cpu_manager.go:414] "containerMap: removing container" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" Oct 31 00:53:17.433017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.432856 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" Oct 31 00:53:17.433017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.432995 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" Oct 31 00:53:17.453839 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.453787 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.464207 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.463370 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m"] Oct 31 00:53:17.576395 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.576344 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.576874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.576408 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.576874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.576543 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.576874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.576578 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.576874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.576687 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj6pm\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-kube-api-access-pj6pm\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.576874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.576716 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.576874 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.576736 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.577162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.576883 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.577162 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.576931 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.678348 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.678307 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.678348 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.678352 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.678651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.678375 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.678651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.678392 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.678651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.678445 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj6pm\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-kube-api-access-pj6pm\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.678651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.678489 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.678651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.678633 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.678920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.678788 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.678920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.678872 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.679583 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.679555 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.679702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.679662 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.679789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.679762 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.679926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.679900 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.679989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.679963 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.681762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.681739 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.682021 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.681998 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.700948 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.700910 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj6pm\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-kube-api-access-pj6pm\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.706097 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.706065 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:17.797912 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:17.797871 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:18.043502 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:18.042426 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:18.191896 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:18.191860 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m"] Oct 31 00:53:18.192543 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:53:18.192512 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf65fb1_003c_46c0_9f78_18365d2c0d63.slice/crio-23a70f36457e4a97765ceeede21e380d0998c4568927d6f94312e08a4f718250 WatchSource:0}: Error finding container 23a70f36457e4a97765ceeede21e380d0998c4568927d6f94312e08a4f718250: Status 404 returned error can't find the container with id 23a70f36457e4a97765ceeede21e380d0998c4568927d6f94312e08a4f718250 Oct 31 00:53:18.195560 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:18.195523 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:18.195677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:18.195600 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:18.195677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:18.195633 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:18.440538 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:18.429814 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:18.981203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:18.981150 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:19.046546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:19.046445 5978 patch_prober.go:29] interesting pod/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:53:19.046741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:19.046580 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:53:19.111703 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:19.111659 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" event={"ID":"cdf65fb1-003c-46c0-9f78-18365d2c0d63","Type":"ContainerStarted","Data":"1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4"} Oct 31 00:53:19.111885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:19.111718 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" event={"ID":"cdf65fb1-003c-46c0-9f78-18365d2c0d63","Type":"ContainerStarted","Data":"23a70f36457e4a97765ceeede21e380d0998c4568927d6f94312e08a4f718250"} Oct 31 00:53:19.173759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:19.173689 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" podStartSLOduration=2.173670879 podStartE2EDuration="2.173670879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:19.16138447 +0000 UTC m=+2016.859498235" watchObservedRunningTime="2026-10-31 00:53:19.173670879 +0000 UTC m=+2016.871784644" Oct 31 00:53:19.427414 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:19.427255 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:19.798306 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:19.798277 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:19.857210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:19.800363 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 00:53:19.857210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:19.800410 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 00:53:20.173718 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:20.173191 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": read tcp 10.134.0.2:50868->10.134.0.146:15021: read: connection reset by peer" start-of-body= Oct 31 00:53:20.173718 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:20.173256 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": read tcp 10.134.0.2:50868->10.134.0.146:15021: read: connection reset by peer" Oct 31 00:53:20.798651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:20.798615 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 00:53:20.798866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:20.798685 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 00:53:21.157503 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.157425 5978 generic.go:356] "Generic (PLEG): container finished" podID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerID="6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386" exitCode=0 Oct 31 00:53:21.157690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.157503 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" event={"ID":"d6314229-18f2-4fbf-b206-0e7e36ac4cdd","Type":"ContainerDied","Data":"6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386"} Oct 31 00:53:21.158161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.158136 5978 scope.go:122] "RemoveContainer" containerID="6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386" Oct 31 00:53:21.160247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.160211 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:21.160363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.160283 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:21.160363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.160324 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:21.395210 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.395174 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:21.590876 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.590835 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:21.759980 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.759403 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:21.798307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.798273 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 00:53:21.798499 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.798331 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 00:53:21.907528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:21.906998 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:22.172968 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.172871 5978 generic.go:356] "Generic (PLEG): container finished" podID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerID="13802e888d9ebe99cf0d832460df845156d9f120864a29ebebcc42a376d39990" exitCode=0 Oct 31 00:53:22.173110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.172996 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" event={"ID":"3a043e49-6ec4-47b9-8323-d49ee320d95c","Type":"ContainerDied","Data":"13802e888d9ebe99cf0d832460df845156d9f120864a29ebebcc42a376d39990"} Oct 31 00:53:22.173110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.173064 5978 scope.go:122] "RemoveContainer" containerID="433a159cdd9a7ab4e0cb271cc4f02c1e063610ec787d9d564137350efff32599" Oct 31 00:53:22.178532 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.177846 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" event={"ID":"d6314229-18f2-4fbf-b206-0e7e36ac4cdd","Type":"ContainerStarted","Data":"da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463"} Oct 31 00:53:22.475596 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.466807 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:53:22.536486 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.536372 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:53:22.536686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.536589 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:53:22.536686 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.536657 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.559144 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-credential-socket\") pod \"3a043e49-6ec4-47b9-8323-d49ee320d95c\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.559212 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a043e49-6ec4-47b9-8323-d49ee320d95c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a043e49-6ec4-47b9-8323-d49ee320d95c-istiod-ca-cert\") pod \"3a043e49-6ec4-47b9-8323-d49ee320d95c\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.559252 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-envoy\") pod \"3a043e49-6ec4-47b9-8323-d49ee320d95c\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.559423 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-certs\") pod \"3a043e49-6ec4-47b9-8323-d49ee320d95c\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.559523 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-token\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-token\") pod \"3a043e49-6ec4-47b9-8323-d49ee320d95c\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.559594 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-data\") pod \"3a043e49-6ec4-47b9-8323-d49ee320d95c\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.559689 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-podinfo\") pod \"3a043e49-6ec4-47b9-8323-d49ee320d95c\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.559757 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-kube-api-access-d5vgl\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-kube-api-access-d5vgl\") pod \"3a043e49-6ec4-47b9-8323-d49ee320d95c\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.559859 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-socket\") pod \"3a043e49-6ec4-47b9-8323-d49ee320d95c\" (UID: \"3a043e49-6ec4-47b9-8323-d49ee320d95c\") " Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.561675 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-credential-socket" pod "3a043e49-6ec4-47b9-8323-d49ee320d95c" (UID: "3a043e49-6ec4-47b9-8323-d49ee320d95c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.561916 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-data" pod "3a043e49-6ec4-47b9-8323-d49ee320d95c" (UID: "3a043e49-6ec4-47b9-8323-d49ee320d95c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.562140 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a043e49-6ec4-47b9-8323-d49ee320d95c-istiod-ca-cert" pod "3a043e49-6ec4-47b9-8323-d49ee320d95c" (UID: "3a043e49-6ec4-47b9-8323-d49ee320d95c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.562682 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-certs" pod "3a043e49-6ec4-47b9-8323-d49ee320d95c" (UID: "3a043e49-6ec4-47b9-8323-d49ee320d95c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:22.567477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.562694 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-socket" pod "3a043e49-6ec4-47b9-8323-d49ee320d95c" (UID: "3a043e49-6ec4-47b9-8323-d49ee320d95c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:22.587142 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.587060 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-podinfo" pod "3a043e49-6ec4-47b9-8323-d49ee320d95c" (UID: "3a043e49-6ec4-47b9-8323-d49ee320d95c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:53:22.587266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.587220 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-envoy" pod "3a043e49-6ec4-47b9-8323-d49ee320d95c" (UID: "3a043e49-6ec4-47b9-8323-d49ee320d95c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:22.587327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.587256 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-kube-api-access-d5vgl" pod "3a043e49-6ec4-47b9-8323-d49ee320d95c" (UID: "3a043e49-6ec4-47b9-8323-d49ee320d95c"). InnerVolumeSpecName "kube-api-access-d5vgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:22.589938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.589907 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-token" pod "3a043e49-6ec4-47b9-8323-d49ee320d95c" (UID: "3a043e49-6ec4-47b9-8323-d49ee320d95c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:22.669323 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.669288 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:22.685478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.683638 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:22.685478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.683680 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a043e49-6ec4-47b9-8323-d49ee320d95c-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:22.685478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.683693 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:22.685478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.683706 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:22.685478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.683719 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:22.685478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.683733 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:22.685478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.683748 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a043e49-6ec4-47b9-8323-d49ee320d95c-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:22.685478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.683762 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5vgl\" (UniqueName: \"kubernetes.io/projected/3a043e49-6ec4-47b9-8323-d49ee320d95c-kube-api-access-d5vgl\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:22.685478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.683776 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a043e49-6ec4-47b9-8323-d49ee320d95c-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:53:22.833537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.821917 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 00:53:22.833537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.821975 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 00:53:22.918498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:22.918195 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:23.398651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.398538 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm"] Oct 31 00:53:23.423506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.399886 5978 cpu_manager.go:414] "containerMap: removing container" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" Oct 31 00:53:23.423506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.399919 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" Oct 31 00:53:23.423506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.399990 5978 cpu_manager.go:414] "containerMap: removing container" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" Oct 31 00:53:23.423506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.400000 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" Oct 31 00:53:23.423506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.400216 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="9171fc04-6c22-4ca2-a581-bbff5c77205a" containerName="istio-proxy" Oct 31 00:53:23.423506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.400238 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" Oct 31 00:53:23.423506 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.400261 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" containerName="istio-proxy" Oct 31 00:53:23.542523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.536447 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" Oct 31 00:53:23.577483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.552242 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:23.577885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.565794 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:53:23.585491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.578053 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:53:23.627504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.623231 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm"] Oct 31 00:53:23.627504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.623283 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms" event={"ID":"3a043e49-6ec4-47b9-8323-d49ee320d95c","Type":"ContainerDied","Data":"47933318921cea5a67e541c9e65fc08f6f091be983f4f1b759d09272af912575"} Oct 31 00:53:23.627504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.623333 5978 scope.go:122] "RemoveContainer" containerID="13802e888d9ebe99cf0d832460df845156d9f120864a29ebebcc42a376d39990" Oct 31 00:53:23.627504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.623534 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.706491 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.701352 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-762753-openshift-default-dockercfg-2crgz\"" Oct 31 00:53:23.769974 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.769920 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-token\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.770194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.770052 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.770194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.770147 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-envoy\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.770194 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.770180 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-credential-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.770437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.770410 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-data\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.770529 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.770489 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istiod-ca-cert\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.770587 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.770556 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz82l\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-kube-api-access-nz82l\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.770653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.770612 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-podinfo\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.770653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.770639 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-certs\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.798797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.798758 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 00:53:23.798983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.798813 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 00:53:23.873583 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.873529 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms"] Oct 31 00:53:23.878319 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.878277 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-data\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.878523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.878339 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istiod-ca-cert\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.878523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.878401 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz82l\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-kube-api-access-nz82l\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.878523 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.878482 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-podinfo\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.878739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.878521 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-certs\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.878739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.878644 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-token\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.878739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.878717 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.878891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.878813 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-envoy\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.878891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.878854 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-credential-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.879528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.879502 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-credential-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.879995 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.879972 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-data\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.880776 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.880755 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istiod-ca-cert\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.883297 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.883264 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3b1945-openshift-default-67fdf96d79-5jpms"] Oct 31 00:53:23.885217 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.883909 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-certs\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.885217 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.884163 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-socket\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.889140 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.889099 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-podinfo\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.889319 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.889300 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-envoy\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.906005 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.905971 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz82l\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-kube-api-access-nz82l\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:23.906842 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:23.906799 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-token\") pod \"e2e-inf-a-762753-openshift-default-85767dbb67-8dghm\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:24.066110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.062210 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:24.102605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.102316 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:24.273229 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.273004 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:24.504767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.504724 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:24.539284 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.539246 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:53:24.550475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.539616 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:53:24.609748 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.609705 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a043e49-6ec4-47b9-8323-d49ee320d95c" path="/var/lib/kubelet/pods/3a043e49-6ec4-47b9-8323-d49ee320d95c/volumes" Oct 31 00:53:24.671274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.670979 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm"] Oct 31 00:53:24.675202 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:53:24.675130 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8d9746_73e2_43f0_bb4f_5b422fcb0ae7.slice/crio-44441d8136374ade5dfa0c5028a956cbc77b027f19b564085878824c3339b521 WatchSource:0}: Error finding container 44441d8136374ade5dfa0c5028a956cbc77b027f19b564085878824c3339b521: Status 404 returned error can't find the container with id 44441d8136374ade5dfa0c5028a956cbc77b027f19b564085878824c3339b521 Oct 31 00:53:24.680920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.680883 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:24.681008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.680961 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:24.681008 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.680991 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:53:24.799312 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.799279 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 00:53:24.799424 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:24.799336 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 00:53:25.550480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:25.550072 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:53:25.550480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:25.550153 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:53:25.697951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:25.697910 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" event={"ID":"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7","Type":"ContainerStarted","Data":"e45d73731f5054cc32759ec98f7b38cf8dba58020f970363d69cab75d5b132e6"} Oct 31 00:53:25.698160 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:25.697976 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" event={"ID":"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7","Type":"ContainerStarted","Data":"44441d8136374ade5dfa0c5028a956cbc77b027f19b564085878824c3339b521"} Oct 31 00:53:25.759257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:25.759134 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" podStartSLOduration=2.759115621 podStartE2EDuration="2.759115621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:25.756354511 +0000 UTC m=+2023.454468276" watchObservedRunningTime="2026-10-31 00:53:25.759115621 +0000 UTC m=+2023.457229387" Oct 31 00:53:25.942816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:25.942769 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:25.943568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:25.943541 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:25.947257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:25.947230 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:53:26.065568 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:26.064099 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:26.116477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:26.108818 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:26.368476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:26.366383 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:26.538514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:26.537931 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:53:26.538514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:26.537995 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:53:26.540885 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:26.539047 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:26.946883 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:26.946675 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:26.947997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:26.947974 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:53:27.536478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:27.536328 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:53:27.536478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:27.536404 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:53:28.433712 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:28.433674 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:28.619401 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:28.619362 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:28.781539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:28.781259 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:53:28.787554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:28.783622 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:53:28.787554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:28.786099 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:53:30.180582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.180539 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:30.326393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.326330 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:30.327151 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.327125 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs"] Oct 31 00:53:30.330923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.330900 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:30.357868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.356734 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs"] Oct 31 00:53:30.358909 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.358881 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-762753-dockercfg-8lqxq\"" Oct 31 00:53:30.359181 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.359165 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-762753\"" Oct 31 00:53:30.527398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.527311 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:30.684940 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.684897 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs\" (UID: \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:30.685245 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.685222 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txrxp\" (UniqueName: \"kubernetes.io/projected/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-kube-api-access-txrxp\") pod \"payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs\" (UID: \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:30.712534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.712399 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m"] Oct 31 00:53:30.742281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.741999 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m"] Oct 31 00:53:30.742281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.742163 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:30.788382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.788280 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txrxp\" (UniqueName: \"kubernetes.io/projected/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-kube-api-access-txrxp\") pod \"payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs\" (UID: \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:30.788382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.788340 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5zkx\" (UniqueName: \"kubernetes.io/projected/67a98371-3e68-422d-87d9-ee4afb582496-kube-api-access-j5zkx\") pod \"payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m\" (UID: \"67a98371-3e68-422d-87d9-ee4afb582496\") " pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:30.788752 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.788725 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a98371-3e68-422d-87d9-ee4afb582496-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m\" (UID: \"67a98371-3e68-422d-87d9-ee4afb582496\") " pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:30.788856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.788830 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs\" (UID: \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:30.790044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.790019 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs\" (UID: \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:30.811195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.811155 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txrxp\" (UniqueName: \"kubernetes.io/projected/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-kube-api-access-txrxp\") pod \"payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs\" (UID: \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:30.890619 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.890578 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a98371-3e68-422d-87d9-ee4afb582496-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m\" (UID: \"67a98371-3e68-422d-87d9-ee4afb582496\") " pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:30.890894 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.890676 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5zkx\" (UniqueName: \"kubernetes.io/projected/67a98371-3e68-422d-87d9-ee4afb582496-kube-api-access-j5zkx\") pod \"payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m\" (UID: \"67a98371-3e68-422d-87d9-ee4afb582496\") " pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:30.891359 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.891329 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a98371-3e68-422d-87d9-ee4afb582496-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m\" (UID: \"67a98371-3e68-422d-87d9-ee4afb582496\") " pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:30.911643 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.911605 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5zkx\" (UniqueName: \"kubernetes.io/projected/67a98371-3e68-422d-87d9-ee4afb582496-kube-api-access-j5zkx\") pod \"payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m\" (UID: \"67a98371-3e68-422d-87d9-ee4afb582496\") " pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:30.987805 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:30.987774 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:31.058979 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:31.058940 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:31.293152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:31.293124 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs"] Oct 31 00:53:31.293929 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:53:31.293904 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fffbd8_b49b_4eaa_ae7c_42132790ca2b.slice/crio-919a721cd4c7c01984bb404ad005a0466777a7bdf9e99dc9df4a9dfd94b69fab WatchSource:0}: Error finding container 919a721cd4c7c01984bb404ad005a0466777a7bdf9e99dc9df4a9dfd94b69fab: Status 404 returned error can't find the container with id 919a721cd4c7c01984bb404ad005a0466777a7bdf9e99dc9df4a9dfd94b69fab Oct 31 00:53:31.382072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:31.382043 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m"] Oct 31 00:53:31.382428 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:53:31.382404 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a98371_3e68_422d_87d9_ee4afb582496.slice/crio-81f2930fe056d1462a99f3f3a92e2a04c2ac8bd1e77acd2946a0a6e61538e39f WatchSource:0}: Error finding container 81f2930fe056d1462a99f3f3a92e2a04c2ac8bd1e77acd2946a0a6e61538e39f: Status 404 returned error can't find the container with id 81f2930fe056d1462a99f3f3a92e2a04c2ac8bd1e77acd2946a0a6e61538e39f Oct 31 00:53:31.990267 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:31.990229 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" event={"ID":"19fffbd8-b49b-4eaa-ae7c-42132790ca2b","Type":"ContainerStarted","Data":"cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4"} Oct 31 00:53:31.990496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:31.990292 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" event={"ID":"19fffbd8-b49b-4eaa-ae7c-42132790ca2b","Type":"ContainerStarted","Data":"919a721cd4c7c01984bb404ad005a0466777a7bdf9e99dc9df4a9dfd94b69fab"} Oct 31 00:53:31.990496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:31.990325 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:31.992259 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:31.992230 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" event={"ID":"67a98371-3e68-422d-87d9-ee4afb582496","Type":"ContainerStarted","Data":"858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f"} Oct 31 00:53:31.992396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:31.992295 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:31.992396 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:31.992319 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" event={"ID":"67a98371-3e68-422d-87d9-ee4afb582496","Type":"ContainerStarted","Data":"81f2930fe056d1462a99f3f3a92e2a04c2ac8bd1e77acd2946a0a6e61538e39f"} Oct 31 00:53:32.028163 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:32.028090 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" podStartSLOduration=2.028068686 podStartE2EDuration="2.028068686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:32.025816764 +0000 UTC m=+2029.723930552" watchObservedRunningTime="2026-10-31 00:53:32.028068686 +0000 UTC m=+2029.726182454" Oct 31 00:53:32.064690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:32.064619 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" podStartSLOduration=2.064600079 podStartE2EDuration="2.064600079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:32.06192452 +0000 UTC m=+2029.760038285" watchObservedRunningTime="2026-10-31 00:53:32.064600079 +0000 UTC m=+2029.762713844" Oct 31 00:53:32.184357 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:32.184315 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:33.184201 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:53:33.184165 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:53:33.842282 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:33.842242 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:34.044153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:34.041317 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:35.822416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:35.822370 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:37.590033 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:37.589991 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:39.310265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:39.310224 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:40.886880 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:40.886794 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:42.444366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:42.444319 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:43.004187 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:43.004148 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 00:53:43.004370 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:43.004223 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 00:53:43.997154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:43.997108 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:45.607901 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:45.602695 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:46.444255 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:46.444212 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:47.320522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:47.320483 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:48.937234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:48.937054 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:50.502067 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:50.501893 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:52.434672 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:52.434628 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:54.435582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:54.435538 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.629479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:54.623733 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.720674 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:54.719623 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.843730 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:54.842548 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9" podUID="847df7f7-0054-4cc9-a296-e64c9f10dd2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 31 00:53:54.874607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:54.874551 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.889487 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:54.886606 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.953996 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:54.953545 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:54.957093 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:54.956319 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Oct 31 00:53:54.986520 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:54.984569 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 00:53:55.041325 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:55.040544 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 00:53:55.081794 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:55.081541 5978 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded" Oct 31 00:53:55.213356 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:55.213317 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:56.866983 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:56.866945 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:58.648509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:53:58.648376 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:00.387300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:00.387259 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:00.532526 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:00.532443 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:02.262111 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.262012 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:02.547244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.547156 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz"] Oct 31 00:54:02.588309 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.588260 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz"] Oct 31 00:54:02.588617 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.588380 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.658399 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.658355 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.658633 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.658577 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.658745 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.658702 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.658815 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.658755 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.658815 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.658784 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvl9\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-kube-api-access-pxvl9\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.658930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.658880 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.659020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.659001 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.659112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.659096 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.659153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.659120 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.760577 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.760535 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.760788 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.760629 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.760788 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.760660 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.760788 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.760681 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvl9\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-kube-api-access-pxvl9\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.760788 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.760719 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.761024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.760863 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.761024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.760963 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.761024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.760987 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.761193 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.761080 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.761193 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.761119 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-data\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.761479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.761428 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-certs\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.761479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.761431 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.761646 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.761605 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-credential-socket\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.761848 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.761826 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istiod-ca-cert\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.763305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.763274 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-envoy\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.764029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.764003 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-podinfo\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.786844 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.786803 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvl9\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-kube-api-access-pxvl9\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.790871 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.790835 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-token\") pod \"e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:02.926768 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:02.926731 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:03.185152 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:54:03.185051 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:54:03.279493 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.279415 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz"] Oct 31 00:54:03.282820 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:54:03.282788 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942cb7e8_aba4_46b2_a7bc_31d3443aa31b.slice/crio-0cfdacf96fa5cc4ff6c3e95b14c417ee754f64fb852b245b94813a28a909f37e WatchSource:0}: Error finding container 0cfdacf96fa5cc4ff6c3e95b14c417ee754f64fb852b245b94813a28a909f37e: Status 404 returned error can't find the container with id 0cfdacf96fa5cc4ff6c3e95b14c417ee754f64fb852b245b94813a28a909f37e Oct 31 00:54:03.285908 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.285866 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:03.286036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.285959 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:03.286036 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.286003 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:03.353355 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.353323 5978 generic.go:356] "Generic (PLEG): container finished" podID="36f3833c-293b-41e4-878a-a36a42207ff2" containerID="9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823" exitCode=0 Oct 31 00:54:03.353562 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.353390 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" event={"ID":"36f3833c-293b-41e4-878a-a36a42207ff2","Type":"ContainerDied","Data":"9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823"} Oct 31 00:54:03.354033 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.354010 5978 scope.go:122] "RemoveContainer" containerID="9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823" Oct 31 00:54:03.355178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.355152 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" event={"ID":"942cb7e8-aba4-46b2-a7bc-31d3443aa31b","Type":"ContainerStarted","Data":"0cfdacf96fa5cc4ff6c3e95b14c417ee754f64fb852b245b94813a28a909f37e"} Oct 31 00:54:03.356626 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.356591 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:03.356705 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.356688 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:03.356759 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:03.356743 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:04.268764 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.268721 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:04.366768 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.366723 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" event={"ID":"942cb7e8-aba4-46b2-a7bc-31d3443aa31b","Type":"ContainerStarted","Data":"c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210"} Oct 31 00:54:04.369364 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.369336 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" event={"ID":"36f3833c-293b-41e4-878a-a36a42207ff2","Type":"ContainerStarted","Data":"6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c"} Oct 31 00:54:04.431216 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.431153 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" podStartSLOduration=2.431130407 podStartE2EDuration="2.431130407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:04.408214324 +0000 UTC m=+2062.106328095" watchObservedRunningTime="2026-10-31 00:54:04.431130407 +0000 UTC m=+2062.129244174" Oct 31 00:54:04.876720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.876684 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:54:04.876940 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.876803 5978 patch_prober.go:29] interesting pod/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:54:04.876940 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.876845 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:54:04.932480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.927858 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:04.949477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.939598 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:54:04.949477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:04.939666 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 00:54:05.877322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:05.877284 5978 patch_prober.go:29] interesting pod/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:54:05.877817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:05.877350 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:54:05.931484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:05.930972 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:54:05.931484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:05.931036 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 00:54:06.107170 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:06.107129 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:06.883480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:06.880380 5978 patch_prober.go:29] interesting pod/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:54:06.883480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:06.880471 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:54:06.941499 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:06.927875 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:54:06.941499 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:06.927963 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 00:54:07.462812 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:07.462761 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:07.759536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:07.758794 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:07.878923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:07.878886 5978 patch_prober.go:29] interesting pod/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:54:07.879726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:07.879684 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:54:07.931402 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:07.931214 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:07.932571 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:07.932495 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:54:07.932796 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:07.932770 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 00:54:08.260536 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:08.260496 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:08.876616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:08.876579 5978 patch_prober.go:29] interesting pod/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:54:08.876788 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:08.876638 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:54:08.931165 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:08.931125 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:54:08.931500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:08.931470 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 00:54:09.759971 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:09.759871 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:09.876611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:09.876576 5978 patch_prober.go:29] interesting pod/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:54:09.876797 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:09.876629 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:54:09.929995 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:09.929953 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 00:54:09.930285 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:09.930257 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 00:54:10.064050 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.063942 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h"] Oct 31 00:54:10.069446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.069416 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.094186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.094153 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h"] Oct 31 00:54:10.096618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.096586 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-762753-serving-cert\"" Oct 31 00:54:10.187431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.187381 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-metrics-certs\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.187683 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.187662 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.187855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.187835 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqwm\" (UniqueName: \"kubernetes.io/projected/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-kube-api-access-2hqwm\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.188119 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.188101 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-maas-api-tls\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.257599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.234720 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:10.289522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.289484 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-metrics-certs\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.289707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.289607 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.289707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.289681 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqwm\" (UniqueName: \"kubernetes.io/projected/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-kube-api-access-2hqwm\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.289819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.289721 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-maas-api-tls\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.290367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.290340 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.292191 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.292143 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-maas-api-tls\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.292280 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.292187 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-metrics-certs\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.336488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.333198 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqwm\" (UniqueName: \"kubernetes.io/projected/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-kube-api-access-2hqwm\") pod \"maas-api-e2e-inf-a-762753-78c86646fd-9wh5h\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.404479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.397686 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:10.404479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.399975 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:10.807727 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:54:10.807688 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a855609_b0fc_400a_a0bf_fe7295d4ac0b.slice/crio-77a4e4014a60f5fb84ef53e433fe0abd68e564d83cf52843ac0a827543daebbd WatchSource:0}: Error finding container 77a4e4014a60f5fb84ef53e433fe0abd68e564d83cf52843ac0a827543daebbd: Status 404 returned error can't find the container with id 77a4e4014a60f5fb84ef53e433fe0abd68e564d83cf52843ac0a827543daebbd Oct 31 00:54:10.808335 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.808301 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h"] Oct 31 00:54:10.935321 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.935282 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:10.935926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.935899 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:10.936989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:10.936962 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:54:11.086878 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:11.086817 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:54:11.090480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:11.088773 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:54:11.094994 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:11.094937 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:54:11.525096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:11.525001 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" event={"ID":"6a855609-b0fc-400a-a0bf-fe7295d4ac0b","Type":"ContainerStarted","Data":"2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc"} Oct 31 00:54:11.525096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:11.525065 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" event={"ID":"6a855609-b0fc-400a-a0bf-fe7295d4ac0b","Type":"ContainerStarted","Data":"77a4e4014a60f5fb84ef53e433fe0abd68e564d83cf52843ac0a827543daebbd"} Oct 31 00:54:11.525096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:11.525096 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:11.614899 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:11.614829 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" podStartSLOduration=1.280523741 podStartE2EDuration="1.614805504s" totalImagesPullingTime="334.281763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:11.562917813 +0000 UTC m=+2069.261031578" watchObservedRunningTime="2026-10-31 00:54:11.614805504 +0000 UTC m=+2069.312919272" Oct 31 00:54:12.669636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:12.669599 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:12.872653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:12.872603 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:14.609734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:14.609698 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:14.763515 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:14.763480 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:15.132472 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:15.128845 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:16.635638 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:16.635589 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:17.549392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:17.549358 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 00:54:18.439961 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:18.439912 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:20.659932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:20.659897 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:21.851469 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:21.851418 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:21.996137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:21.996099 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:22.892136 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:22.892101 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:24.810598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:24.807569 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:25.371240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:25.371204 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:26.241816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:26.241778 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:26.438542 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:26.438447 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:26.705671 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:26.705633 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:28.310231 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:28.310188 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:29.909250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:29.909209 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:31.533781 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:31.533742 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:33.185432 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:54:33.185392 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:54:33.206617 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:33.206581 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:34.833480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:34.833430 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:36.617430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:36.617385 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:38.419495 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:38.419418 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:40.039918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:40.039862 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:41.606933 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:41.606874 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:42.098660 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:42.098623 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:43.668600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:43.668552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:43.891361 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:43.891310 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:45.902514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:45.899955 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:46.568001 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:46.567964 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:47.862271 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:47.862236 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:49.650720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:49.650678 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:49.811363 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:49.811324 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:50.338059 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:50.338025 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:50.347749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:50.347723 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:54:50.348555 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:50.348533 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:54:50.926311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:50.919411 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr"] Oct 31 00:54:50.926311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:50.919747 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" containerID="cri-o://c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed" gracePeriod=30 Oct 31 00:54:51.040156 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:51.040097 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:51.236530 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:51.231050 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:51.891206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:51.891176 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:53.962240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:53.962201 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:55.478055 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:55.478014 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:55.680577 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:55.680539 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:55.867250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:55.867164 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:56.073931 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.073897 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:56.338754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.338716 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:54:56.430599 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.430509 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-credential-socket\") pod \"55211eb4-7459-4e02-a513-087e43450b5b\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " Oct 31 00:54:56.430879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.430596 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-envoy\") pod \"55211eb4-7459-4e02-a513-087e43450b5b\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " Oct 31 00:54:56.430879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.430672 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-socket\") pod \"55211eb4-7459-4e02-a513-087e43450b5b\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " Oct 31 00:54:56.430879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.430713 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-certs\") pod \"55211eb4-7459-4e02-a513-087e43450b5b\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " Oct 31 00:54:56.430879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.430764 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-istio-token\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-istio-token\") pod \"55211eb4-7459-4e02-a513-087e43450b5b\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " Oct 31 00:54:56.430879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.430852 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-credential-socket" pod "55211eb4-7459-4e02-a513-087e43450b5b" (UID: "55211eb4-7459-4e02-a513-087e43450b5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.431179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.430986 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55211eb4-7459-4e02-a513-087e43450b5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55211eb4-7459-4e02-a513-087e43450b5b-istiod-ca-cert\") pod \"55211eb4-7459-4e02-a513-087e43450b5b\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " Oct 31 00:54:56.431179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.431007 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-certs" pod "55211eb4-7459-4e02-a513-087e43450b5b" (UID: "55211eb4-7459-4e02-a513-087e43450b5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.431179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.431024 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55211eb4-7459-4e02-a513-087e43450b5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55211eb4-7459-4e02-a513-087e43450b5b-istio-podinfo\") pod \"55211eb4-7459-4e02-a513-087e43450b5b\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " Oct 31 00:54:56.431179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.431108 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-data\") pod \"55211eb4-7459-4e02-a513-087e43450b5b\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " Oct 31 00:54:56.431179 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.431156 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-kube-api-access-p79qp\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-kube-api-access-p79qp\") pod \"55211eb4-7459-4e02-a513-087e43450b5b\" (UID: \"55211eb4-7459-4e02-a513-087e43450b5b\") " Oct 31 00:54:56.431446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.431282 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55211eb4-7459-4e02-a513-087e43450b5b-istiod-ca-cert" pod "55211eb4-7459-4e02-a513-087e43450b5b" (UID: "55211eb4-7459-4e02-a513-087e43450b5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:56.431446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.431297 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-socket" pod "55211eb4-7459-4e02-a513-087e43450b5b" (UID: "55211eb4-7459-4e02-a513-087e43450b5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.431607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.431568 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-data" pod "55211eb4-7459-4e02-a513-087e43450b5b" (UID: "55211eb4-7459-4e02-a513-087e43450b5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.432241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.432218 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.432241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.432240 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.432398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.432250 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55211eb4-7459-4e02-a513-087e43450b5b-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.432398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.432259 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.432398 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.432271 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.433426 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.433396 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-envoy" pod "55211eb4-7459-4e02-a513-087e43450b5b" (UID: "55211eb4-7459-4e02-a513-087e43450b5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.433701 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.433675 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-istio-token" pod "55211eb4-7459-4e02-a513-087e43450b5b" (UID: "55211eb4-7459-4e02-a513-087e43450b5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:56.433988 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.433967 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55211eb4-7459-4e02-a513-087e43450b5b-istio-podinfo" pod "55211eb4-7459-4e02-a513-087e43450b5b" (UID: "55211eb4-7459-4e02-a513-087e43450b5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:56.434935 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.434916 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-kube-api-access-p79qp" pod "55211eb4-7459-4e02-a513-087e43450b5b" (UID: "55211eb4-7459-4e02-a513-087e43450b5b"). InnerVolumeSpecName "kube-api-access-p79qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:56.534535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.534493 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55211eb4-7459-4e02-a513-087e43450b5b-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.534535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.534524 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p79qp\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-kube-api-access-p79qp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.534535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.534535 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55211eb4-7459-4e02-a513-087e43450b5b-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.534535 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.534544 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55211eb4-7459-4e02-a513-087e43450b5b-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:54:56.866475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.864596 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:56.950881 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.950606 5978 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:54:56.951278 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:56.950928 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:54:57.068131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.068083 5978 generic.go:356] "Generic (PLEG): container finished" podID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerID="e45d73731f5054cc32759ec98f7b38cf8dba58020f970363d69cab75d5b132e6" exitCode=0 Oct 31 00:54:57.068408 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.068148 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" event={"ID":"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7","Type":"ContainerDied","Data":"e45d73731f5054cc32759ec98f7b38cf8dba58020f970363d69cab75d5b132e6"} Oct 31 00:54:57.068929 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.068909 5978 scope.go:122] "RemoveContainer" containerID="e45d73731f5054cc32759ec98f7b38cf8dba58020f970363d69cab75d5b132e6" Oct 31 00:54:57.070999 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.070973 5978 generic.go:356] "Generic (PLEG): container finished" podID="55211eb4-7459-4e02-a513-087e43450b5b" containerID="c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed" exitCode=0 Oct 31 00:54:57.071129 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.071059 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" Oct 31 00:54:57.071129 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.071064 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" event={"ID":"55211eb4-7459-4e02-a513-087e43450b5b","Type":"ContainerDied","Data":"c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed"} Oct 31 00:54:57.071254 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.071136 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr" event={"ID":"55211eb4-7459-4e02-a513-087e43450b5b","Type":"ContainerDied","Data":"3548f65974234ba049fe69cd5eb3c9651b273440afa29e91330562ad27c7894a"} Oct 31 00:54:57.071254 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.071170 5978 scope.go:122] "RemoveContainer" containerID="c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed" Oct 31 00:54:57.071645 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.071609 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:57.071772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.071735 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:57.071849 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.071788 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:54:57.088504 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.088479 5978 scope.go:122] "RemoveContainer" containerID="c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed" Oct 31 00:54:57.088826 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:54:57.088807 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed\": container with ID starting with c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed not found: ID does not exist" containerID="c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed" Oct 31 00:54:57.088892 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.088836 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed"} err="failed to get container status \"c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed\": rpc error: code = NotFound desc = could not find container \"c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed\": container with ID starting with c593ca7b029407f239b1e26dd9adc3b2069457273072f263d14f727abd2738ed not found: ID does not exist" Oct 31 00:54:57.166112 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.166078 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr"] Oct 31 00:54:57.170772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.170740 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-kzfbr"] Oct 31 00:54:57.291478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.290312 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:57.625119 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.625083 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:57.782378 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:57.782342 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:58.121053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:58.115863 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" event={"ID":"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7","Type":"ContainerStarted","Data":"399f3add74301b379dc9d42a63f557e18257d95caee08f2df438074d45be52ee"} Oct 31 00:54:58.439810 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:58.439726 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55211eb4-7459-4e02-a513-087e43450b5b" path="/var/lib/kubelet/pods/55211eb4-7459-4e02-a513-087e43450b5b/volumes" Oct 31 00:54:58.809767 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:58.808712 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:59.063961 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:59.063878 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:54:59.064152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:59.064127 5978 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:54:59.064203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:59.064182 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:54:59.295064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:54:59.295032 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:00.064021 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:00.063982 5978 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:55:00.064431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:00.064039 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:55:00.837207 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:00.837171 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:01.028478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:01.021521 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:01.074416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:01.072228 5978 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:55:01.074416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:01.072302 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:55:01.203083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:01.202987 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:01.442531 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:01.441530 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:01.643738 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:01.643685 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:01.850010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:01.849972 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:02.064087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:02.064007 5978 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:55:02.064087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:02.064062 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:55:03.062087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.062053 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:03.063724 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.063702 5978 patch_prober.go:29] interesting pod/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:55:03.063808 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.063758 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:55:03.187331 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:55:03.187290 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:55:03.267742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.267701 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj"] Oct 31 00:55:03.275366 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.275307 5978 cpu_manager.go:414] "containerMap: removing container" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" Oct 31 00:55:03.275710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.275676 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" Oct 31 00:55:03.276198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.276179 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="55211eb4-7459-4e02-a513-087e43450b5b" containerName="istio-proxy" Oct 31 00:55:03.284404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.284367 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:03.316563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.309807 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:03.316563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.311278 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8732d41a-8b1b-49dc-b421-1df166cf5a74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj\" (UID: \"8732d41a-8b1b-49dc-b421-1df166cf5a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:03.316563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.311351 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skt2c\" (UniqueName: \"kubernetes.io/projected/8732d41a-8b1b-49dc-b421-1df166cf5a74-kube-api-access-skt2c\") pod \"payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj\" (UID: \"8732d41a-8b1b-49dc-b421-1df166cf5a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:03.393477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.387872 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9de7f5\"" Oct 31 00:55:03.399583 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.394888 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj"] Oct 31 00:55:03.562475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.534862 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8732d41a-8b1b-49dc-b421-1df166cf5a74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj\" (UID: \"8732d41a-8b1b-49dc-b421-1df166cf5a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:03.562475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.534949 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt2c\" (UniqueName: \"kubernetes.io/projected/8732d41a-8b1b-49dc-b421-1df166cf5a74-kube-api-access-skt2c\") pod \"payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj\" (UID: \"8732d41a-8b1b-49dc-b421-1df166cf5a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:03.562475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.536437 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8732d41a-8b1b-49dc-b421-1df166cf5a74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj\" (UID: \"8732d41a-8b1b-49dc-b421-1df166cf5a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:03.574563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.571893 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9de7f5-dockercfg-66gbr\"" Oct 31 00:55:03.686785 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.686749 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:03.689338 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.689312 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skt2c\" (UniqueName: \"kubernetes.io/projected/8732d41a-8b1b-49dc-b421-1df166cf5a74-kube-api-access-skt2c\") pod \"payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj\" (UID: \"8732d41a-8b1b-49dc-b421-1df166cf5a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:03.986324 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:03.986026 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:04.182015 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:04.181982 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:55:04.183020 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:04.182995 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:55:04.185480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:04.185439 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 00:55:04.341534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:04.327389 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj"] Oct 31 00:55:05.199288 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:05.199247 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" event={"ID":"8732d41a-8b1b-49dc-b421-1df166cf5a74","Type":"ContainerStarted","Data":"7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2"} Oct 31 00:55:05.199769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:05.199311 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" event={"ID":"8732d41a-8b1b-49dc-b421-1df166cf5a74","Type":"ContainerStarted","Data":"6ef5c68c25045005e318f8ad2e67d14b19f82c77727cfaa16d94bce85df5587e"} Oct 31 00:55:05.199769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:05.199334 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:05.240079 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:05.240020 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" podStartSLOduration=2.240004302 podStartE2EDuration="2.240004302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:05.239602151 +0000 UTC m=+2122.937715928" watchObservedRunningTime="2026-10-31 00:55:05.240004302 +0000 UTC m=+2122.938118068" Oct 31 00:55:05.671910 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:05.671878 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:05.830742 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:05.830710 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:05.989920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:05.989812 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:07.910203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:07.910035 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:08.066324 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:08.066283 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:08.313528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:08.298444 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:09.713404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:09.713350 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:11.267817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:11.267774 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:13.136233 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:13.136189 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:14.410647 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:14.410606 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:14.572173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:14.572133 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.002660 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:15.002620 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.566898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:15.566862 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.714906 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:15.714858 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:16.213022 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:16.212992 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 00:55:16.691760 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:16.691713 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:18.363478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:18.353765 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:20.015517 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:20.015435 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:21.602714 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:21.602675 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:23.457937 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:23.457814 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:24.857670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:24.857627 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:25.617804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:25.617763 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:25.780169 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:25.780132 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:25.928329 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:25.928232 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:26.330954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:26.330910 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:27.625480 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:27.623104 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:29.670422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:29.670383 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:31.375570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:31.375520 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:33.064717 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:33.064375 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:33.184153 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:55:33.184117 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:55:33.346149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:33.346113 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:33.495601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:33.495561 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:33.723076 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:33.722992 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:34.363862 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.363826 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk"] Oct 31 00:55:34.369601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.369576 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.389351 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.389319 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk"] Oct 31 00:55:34.394438 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.393250 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:55:34.394438 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.393414 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:55:34.487153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.487108 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.487367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.487344 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjst\" (UniqueName: \"kubernetes.io/projected/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-kube-api-access-vsjst\") pod \"maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.487414 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.487400 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.589422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.589383 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjst\" (UniqueName: \"kubernetes.io/projected/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-kube-api-access-vsjst\") pod \"maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.589422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.589427 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.589658 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.589644 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.590092 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.590070 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.592056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.592037 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.610886 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.610855 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjst\" (UniqueName: \"kubernetes.io/projected/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-kube-api-access-vsjst\") pod \"maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.684327 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.684242 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:34.981226 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:34.981122 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:35.012340 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:35.012303 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk"] Oct 31 00:55:35.022953 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:55:35.022909 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27bcad0_d9a7_4346_9c09_8f34040c9fc7.slice/crio-ac16ae8e687adfe9af020a1ce1bb597b66fe8cc3babf70820f6d464bb7ecab3b WatchSource:0}: Error finding container ac16ae8e687adfe9af020a1ce1bb597b66fe8cc3babf70820f6d464bb7ecab3b: Status 404 returned error can't find the container with id ac16ae8e687adfe9af020a1ce1bb597b66fe8cc3babf70820f6d464bb7ecab3b Oct 31 00:55:35.533247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:35.533206 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" event={"ID":"d27bcad0-d9a7-4346-9c09-8f34040c9fc7","Type":"ContainerStarted","Data":"dca06491f6bf6784214dc287c9604be2c763f442dadc70fe2d5e3b4c9259e2f5"} Oct 31 00:55:35.533679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:35.533276 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" event={"ID":"d27bcad0-d9a7-4346-9c09-8f34040c9fc7","Type":"ContainerStarted","Data":"ac16ae8e687adfe9af020a1ce1bb597b66fe8cc3babf70820f6d464bb7ecab3b"} Oct 31 00:55:35.580719 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:35.580602 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" podStartSLOduration=1.580588514 podStartE2EDuration="1.580588514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:35.576068128 +0000 UTC m=+2153.274181893" watchObservedRunningTime="2026-10-31 00:55:35.580588514 +0000 UTC m=+2153.278702279" Oct 31 00:55:36.543819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:36.543781 5978 generic.go:356] "Generic (PLEG): container finished" podID="d27bcad0-d9a7-4346-9c09-8f34040c9fc7" containerID="dca06491f6bf6784214dc287c9604be2c763f442dadc70fe2d5e3b4c9259e2f5" exitCode=0 Oct 31 00:55:36.544240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:36.543873 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" event={"ID":"d27bcad0-d9a7-4346-9c09-8f34040c9fc7","Type":"ContainerDied","Data":"dca06491f6bf6784214dc287c9604be2c763f442dadc70fe2d5e3b4c9259e2f5"} Oct 31 00:55:36.815678 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:36.815586 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:36.969266 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:36.969180 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:37.782500 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:37.782465 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:37.975525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:37.972098 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-xks-ca\") pod \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " Oct 31 00:55:37.975525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:37.972242 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-kube-api-access-vsjst\" (UniqueName: \"kubernetes.io/projected/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-kube-api-access-vsjst\") pod \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " Oct 31 00:55:37.975525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:37.972440 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-openshift-ca\") pod \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\" (UID: \"d27bcad0-d9a7-4346-9c09-8f34040c9fc7\") " Oct 31 00:55:37.990474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:37.982872 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-openshift-ca" pod "d27bcad0-d9a7-4346-9c09-8f34040c9fc7" (UID: "d27bcad0-d9a7-4346-9c09-8f34040c9fc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:38.000475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:37.994032 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:55:38.042477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:38.032964 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-xks-ca" pod "d27bcad0-d9a7-4346-9c09-8f34040c9fc7" (UID: "d27bcad0-d9a7-4346-9c09-8f34040c9fc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:38.042477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:38.033507 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-kube-api-access-vsjst" pod "d27bcad0-d9a7-4346-9c09-8f34040c9fc7" (UID: "d27bcad0-d9a7-4346-9c09-8f34040c9fc7"). InnerVolumeSpecName "kube-api-access-vsjst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:38.096202 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:38.096156 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:55:38.096202 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:38.096193 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsjst\" (UniqueName: \"kubernetes.io/projected/d27bcad0-d9a7-4346-9c09-8f34040c9fc7-kube-api-access-vsjst\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:55:38.574116 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:38.574085 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" Oct 31 00:55:38.574318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:38.574183 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk" event={"ID":"d27bcad0-d9a7-4346-9c09-8f34040c9fc7","Type":"ContainerDied","Data":"ac16ae8e687adfe9af020a1ce1bb597b66fe8cc3babf70820f6d464bb7ecab3b"} Oct 31 00:55:38.574318 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:38.574241 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac16ae8e687adfe9af020a1ce1bb597b66fe8cc3babf70820f6d464bb7ecab3b" Oct 31 00:55:38.843284 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:38.840865 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:40.479067 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:40.479026 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:41.122551 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:41.122512 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:42.076513 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:42.076471 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:42.245873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:42.245819 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:42.384753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:42.384650 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:43.174134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:43.174096 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:43.847516 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:43.847476 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:44.915692 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:44.915658 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g"] Oct 31 00:55:45.127204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:45.127157 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:45.276835 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:45.276801 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:45.654476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:45.644759 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:46.693400 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:46.693355 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:47.342158 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.342122 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4"] Oct 31 00:55:47.343311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.343283 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d27bcad0-d9a7-4346-9c09-8f34040c9fc7" containerName="revoke-keys" Oct 31 00:55:47.343311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.343310 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27bcad0-d9a7-4346-9c09-8f34040c9fc7" containerName="revoke-keys" Oct 31 00:55:47.343552 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.343535 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27bcad0-d9a7-4346-9c09-8f34040c9fc7" containerName="revoke-keys" Oct 31 00:55:47.348183 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.348162 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.370082 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.370049 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4"] Oct 31 00:55:47.375572 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.375542 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-72120c-serving-cert\"" Oct 31 00:55:47.454630 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.454590 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-metrics-certs\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.454791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.454667 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b4397e-f00d-402e-9cb2-161b2e8396bd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.454865 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.454831 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpvbl\" (UniqueName: \"kubernetes.io/projected/a1b4397e-f00d-402e-9cb2-161b2e8396bd-kube-api-access-rpvbl\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.454903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.454859 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-maas-api-tls\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.515192 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.515159 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:47.556938 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.556901 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b4397e-f00d-402e-9cb2-161b2e8396bd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.557153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.557005 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpvbl\" (UniqueName: \"kubernetes.io/projected/a1b4397e-f00d-402e-9cb2-161b2e8396bd-kube-api-access-rpvbl\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.557153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.557030 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-maas-api-tls\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.557292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.557273 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-metrics-certs\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.557670 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.557634 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b4397e-f00d-402e-9cb2-161b2e8396bd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.559687 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.559664 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-metrics-certs\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.559853 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.559826 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-maas-api-tls\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.578653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.578613 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpvbl\" (UniqueName: \"kubernetes.io/projected/a1b4397e-f00d-402e-9cb2-161b2e8396bd-kube-api-access-rpvbl\") pod \"maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.665857 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.665743 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:47.689611 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:47.689575 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:48.013684 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:48.013640 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4"] Oct 31 00:55:48.014309 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:55:48.014280 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b4397e_f00d_402e_9cb2_161b2e8396bd.slice/crio-01814ecdbe7c5f374945e6037c707535ff1b2ae751139e5e2edd64338838a0ae WatchSource:0}: Error finding container 01814ecdbe7c5f374945e6037c707535ff1b2ae751139e5e2edd64338838a0ae: Status 404 returned error can't find the container with id 01814ecdbe7c5f374945e6037c707535ff1b2ae751139e5e2edd64338838a0ae Oct 31 00:55:48.744702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:48.744591 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" event={"ID":"a1b4397e-f00d-402e-9cb2-161b2e8396bd","Type":"ContainerStarted","Data":"14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436"} Oct 31 00:55:48.744702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:48.744667 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" event={"ID":"a1b4397e-f00d-402e-9cb2-161b2e8396bd","Type":"ContainerStarted","Data":"01814ecdbe7c5f374945e6037c707535ff1b2ae751139e5e2edd64338838a0ae"} Oct 31 00:55:48.744702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:48.744697 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:48.786091 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:48.785915 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" podStartSLOduration=1.4690594940000001 podStartE2EDuration="1.785895615s" totalImagesPullingTime="316.836121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:48.783230743 +0000 UTC m=+2166.481344508" watchObservedRunningTime="2026-10-31 00:55:48.785895615 +0000 UTC m=+2166.484009382" Oct 31 00:55:49.128436 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:49.128400 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:49.916272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:49.916227 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" podUID="47678034-2289-4238-8e31-074e2c1e5c78" containerName="payload-processing" containerID="cri-o://1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf" gracePeriod=25 Oct 31 00:55:49.922310 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:49.922275 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.100328 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.100287 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.281275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.281239 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.293143 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.293108 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:55:50.389958 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.389902 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47678034-2289-4238-8e31-074e2c1e5c78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47678034-2289-4238-8e31-074e2c1e5c78-plugins-config-volume\") pod \"47678034-2289-4238-8e31-074e2c1e5c78\" (UID: \"47678034-2289-4238-8e31-074e2c1e5c78\") " Oct 31 00:55:50.390222 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.390089 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47678034-2289-4238-8e31-074e2c1e5c78-kube-api-access-44gnp\" (UniqueName: \"kubernetes.io/projected/47678034-2289-4238-8e31-074e2c1e5c78-kube-api-access-44gnp\") pod \"47678034-2289-4238-8e31-074e2c1e5c78\" (UID: \"47678034-2289-4238-8e31-074e2c1e5c78\") " Oct 31 00:55:50.391137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.390940 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47678034-2289-4238-8e31-074e2c1e5c78-plugins-config-volume" pod "47678034-2289-4238-8e31-074e2c1e5c78" (UID: "47678034-2289-4238-8e31-074e2c1e5c78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:50.391819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.391794 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47678034-2289-4238-8e31-074e2c1e5c78-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:55:50.393420 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.393391 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47678034-2289-4238-8e31-074e2c1e5c78-kube-api-access-44gnp" pod "47678034-2289-4238-8e31-074e2c1e5c78" (UID: "47678034-2289-4238-8e31-074e2c1e5c78"). InnerVolumeSpecName "kube-api-access-44gnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:50.431915 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.431857 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.493987 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.493899 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44gnp\" (UniqueName: \"kubernetes.io/projected/47678034-2289-4238-8e31-074e2c1e5c78-kube-api-access-44gnp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:55:50.662287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.662242 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.764866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.764763 5978 generic.go:356] "Generic (PLEG): container finished" podID="47678034-2289-4238-8e31-074e2c1e5c78" containerID="1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf" exitCode=0 Oct 31 00:55:50.764866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.764830 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" Oct 31 00:55:50.765098 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.764871 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" event={"ID":"47678034-2289-4238-8e31-074e2c1e5c78","Type":"ContainerDied","Data":"1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf"} Oct 31 00:55:50.765098 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.764961 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g" event={"ID":"47678034-2289-4238-8e31-074e2c1e5c78","Type":"ContainerDied","Data":"5f4bfae22f3861b16ca57630d1e4df57784ab4b103bbf2f227596ccb758a6143"} Oct 31 00:55:50.765098 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.765021 5978 scope.go:122] "RemoveContainer" containerID="1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf" Oct 31 00:55:50.779109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.779084 5978 scope.go:122] "RemoveContainer" containerID="1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf" Oct 31 00:55:50.779534 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:55:50.779500 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf\": container with ID starting with 1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf not found: ID does not exist" containerID="1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf" Oct 31 00:55:50.779691 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.779543 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf"} err="failed to get container status \"1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf\": rpc error: code = NotFound desc = could not find container \"1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf\": container with ID starting with 1f07c5da1d873d5a2af6502922398d28bedeabc2e8e2b9f2e11b3eaf092c6adf not found: ID does not exist" Oct 31 00:55:50.826560 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.826526 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g"] Oct 31 00:55:50.831510 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:50.831474 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdb61f-764449d9c5-86l2g"] Oct 31 00:55:52.013951 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:52.013909 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:52.428322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:52.428289 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47678034-2289-4238-8e31-074e2c1e5c78" path="/var/lib/kubelet/pods/47678034-2289-4238-8e31-074e2c1e5c78/volumes" Oct 31 00:55:53.834012 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:53.824336 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:54.759973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:54.759943 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 00:55:56.491474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:56.486904 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:57.958969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:57.958929 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:58.173368 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.173325 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:58.261056 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.260967 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz"] Oct 31 00:55:58.261606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.261549 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" containerID="cri-o://c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210" gracePeriod=30 Oct 31 00:55:58.269013 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.268984 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg"] Oct 31 00:55:58.269301 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.269274 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" containerID="cri-o://da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463" gracePeriod=30 Oct 31 00:55:58.285903 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.285873 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m"] Oct 31 00:55:58.286071 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.286053 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" containerID="cri-o://1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4" gracePeriod=30 Oct 31 00:55:58.370849 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.370804 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:58.623868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.623819 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:58.785135 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.785100 5978 patch_prober.go:29] interesting pod/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:55:58.785304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:58.785175 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:55:59.303626 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:59.303595 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:59.812377 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:59.812340 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:59.990507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:55:59.988175 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.545149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:00.542869 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:02.024161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:02.024066 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:02.375572 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:02.375534 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:03.183312 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:56:03.183278 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:56:03.451873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.451788 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:03.610772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.609851 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:56:03.635773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.635739 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:56:03.664084 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.664056 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:56:03.682602 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.682561 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:03.774185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.774098 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-envoy\") pod \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " Oct 31 00:56:03.774185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.774145 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-podinfo\") pod \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " Oct 31 00:56:03.774185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.774176 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-certs\") pod \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " Oct 31 00:56:03.774546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.774200 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istiod-ca-cert\") pod \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " Oct 31 00:56:03.774546 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.774234 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-token\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-token\") pod \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " Oct 31 00:56:03.774679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.774533 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-certs" pod "942cb7e8-aba4-46b2-a7bc-31d3443aa31b" (UID: "942cb7e8-aba4-46b2-a7bc-31d3443aa31b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.774679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.774645 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istiod-ca-cert" pod "942cb7e8-aba4-46b2-a7bc-31d3443aa31b" (UID: "942cb7e8-aba4-46b2-a7bc-31d3443aa31b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:03.775773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.775745 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-kube-api-access-pj6pm\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-kube-api-access-pj6pm\") pod \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " Oct 31 00:56:03.775923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.775838 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-token\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-token\") pod \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " Oct 31 00:56:03.775923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.775872 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-certs\") pod \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " Oct 31 00:56:03.775923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.775901 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istiod-ca-cert\") pod \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " Oct 31 00:56:03.777690 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.777666 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istiod-ca-cert" pod "cdf65fb1-003c-46c0-9f78-18365d2c0d63" (UID: "cdf65fb1-003c-46c0-9f78-18365d2c0d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:03.778623 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.778597 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-certs" pod "cdf65fb1-003c-46c0-9f78-18365d2c0d63" (UID: "cdf65fb1-003c-46c0-9f78-18365d2c0d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.778825 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.778687 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-kube-api-access-z8s4s\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-kube-api-access-z8s4s\") pod \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " Oct 31 00:56:03.778825 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.778721 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-credential-socket\") pod \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " Oct 31 00:56:03.778825 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.778792 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-podinfo\") pod \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " Oct 31 00:56:03.779002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.778873 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-token\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-token\") pod \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " Oct 31 00:56:03.779002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.778907 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-socket\") pod \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " Oct 31 00:56:03.779002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.778954 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-data\") pod \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " Oct 31 00:56:03.779155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.779026 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-data\") pod \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " Oct 31 00:56:03.779155 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.779064 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istiod-ca-cert\") pod \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " Oct 31 00:56:03.779699 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.779674 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-kube-api-access-pj6pm" pod "cdf65fb1-003c-46c0-9f78-18365d2c0d63" (UID: "cdf65fb1-003c-46c0-9f78-18365d2c0d63"). InnerVolumeSpecName "kube-api-access-pj6pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.783764 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.780289 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-data" pod "942cb7e8-aba4-46b2-a7bc-31d3443aa31b" (UID: "942cb7e8-aba4-46b2-a7bc-31d3443aa31b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.783943 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.782051 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-credential-socket" pod "942cb7e8-aba4-46b2-a7bc-31d3443aa31b" (UID: "942cb7e8-aba4-46b2-a7bc-31d3443aa31b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.783943 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.782385 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-socket" pod "cdf65fb1-003c-46c0-9f78-18365d2c0d63" (UID: "cdf65fb1-003c-46c0-9f78-18365d2c0d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.783943 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.782430 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-data" pod "cdf65fb1-003c-46c0-9f78-18365d2c0d63" (UID: "cdf65fb1-003c-46c0-9f78-18365d2c0d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.783943 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.782833 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istiod-ca-cert" pod "d6314229-18f2-4fbf-b206-0e7e36ac4cdd" (UID: "d6314229-18f2-4fbf-b206-0e7e36ac4cdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:03.783943 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.783892 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-certs\") pod \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " Oct 31 00:56:03.784636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.783966 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-podinfo\") pod \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " Oct 31 00:56:03.784636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.784012 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-envoy\") pod \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " Oct 31 00:56:03.784636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.784037 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-data\") pod \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " Oct 31 00:56:03.784636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.784058 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-credential-socket\") pod \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\" (UID: \"cdf65fb1-003c-46c0-9f78-18365d2c0d63\") " Oct 31 00:56:03.784636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.784107 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-envoy\") pod \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " Oct 31 00:56:03.784636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.784161 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-socket\") pod \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " Oct 31 00:56:03.784636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.784200 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-credential-socket\") pod \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " Oct 31 00:56:03.784636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.784256 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-socket\") pod \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\" (UID: \"d6314229-18f2-4fbf-b206-0e7e36ac4cdd\") " Oct 31 00:56:03.784636 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.784289 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-kube-api-access-pxvl9\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-kube-api-access-pxvl9\") pod \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\" (UID: \"942cb7e8-aba4-46b2-a7bc-31d3443aa31b\") " Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785273 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785300 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785314 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785325 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785337 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj6pm\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-kube-api-access-pj6pm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785349 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785363 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785375 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785386 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785399 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785396 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-data" pod "d6314229-18f2-4fbf-b206-0e7e36ac4cdd" (UID: "d6314229-18f2-4fbf-b206-0e7e36ac4cdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785599 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-credential-socket" pod "cdf65fb1-003c-46c0-9f78-18365d2c0d63" (UID: "cdf65fb1-003c-46c0-9f78-18365d2c0d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785608 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-credential-socket" pod "d6314229-18f2-4fbf-b206-0e7e36ac4cdd" (UID: "d6314229-18f2-4fbf-b206-0e7e36ac4cdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785784 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-socket" pod "d6314229-18f2-4fbf-b206-0e7e36ac4cdd" (UID: "d6314229-18f2-4fbf-b206-0e7e36ac4cdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785842 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-podinfo" pod "942cb7e8-aba4-46b2-a7bc-31d3443aa31b" (UID: "942cb7e8-aba4-46b2-a7bc-31d3443aa31b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.785948 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-kube-api-access-z8s4s" pod "d6314229-18f2-4fbf-b206-0e7e36ac4cdd" (UID: "d6314229-18f2-4fbf-b206-0e7e36ac4cdd"). InnerVolumeSpecName "kube-api-access-z8s4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.786070 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-token" pod "cdf65fb1-003c-46c0-9f78-18365d2c0d63" (UID: "cdf65fb1-003c-46c0-9f78-18365d2c0d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.786126 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-socket" pod "942cb7e8-aba4-46b2-a7bc-31d3443aa31b" (UID: "942cb7e8-aba4-46b2-a7bc-31d3443aa31b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.786130 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-token" pod "942cb7e8-aba4-46b2-a7bc-31d3443aa31b" (UID: "942cb7e8-aba4-46b2-a7bc-31d3443aa31b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.786200 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.786174 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-podinfo" pod "d6314229-18f2-4fbf-b206-0e7e36ac4cdd" (UID: "d6314229-18f2-4fbf-b206-0e7e36ac4cdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:03.789165 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.786216 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-envoy" pod "d6314229-18f2-4fbf-b206-0e7e36ac4cdd" (UID: "d6314229-18f2-4fbf-b206-0e7e36ac4cdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.789165 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.788874 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-podinfo" pod "cdf65fb1-003c-46c0-9f78-18365d2c0d63" (UID: "cdf65fb1-003c-46c0-9f78-18365d2c0d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:03.789322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.789286 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-envoy" pod "942cb7e8-aba4-46b2-a7bc-31d3443aa31b" (UID: "942cb7e8-aba4-46b2-a7bc-31d3443aa31b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.789441 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.789426 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-certs" pod "d6314229-18f2-4fbf-b206-0e7e36ac4cdd" (UID: "d6314229-18f2-4fbf-b206-0e7e36ac4cdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.789829 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.789809 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-token" pod "d6314229-18f2-4fbf-b206-0e7e36ac4cdd" (UID: "d6314229-18f2-4fbf-b206-0e7e36ac4cdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.790841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.790785 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-kube-api-access-pxvl9" pod "942cb7e8-aba4-46b2-a7bc-31d3443aa31b" (UID: "942cb7e8-aba4-46b2-a7bc-31d3443aa31b"). InnerVolumeSpecName "kube-api-access-pxvl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.791177 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.791157 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-envoy" pod "cdf65fb1-003c-46c0-9f78-18365d2c0d63" (UID: "cdf65fb1-003c-46c0-9f78-18365d2c0d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.887044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887009 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887039 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887044 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887050 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxvl9\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-kube-api-access-pxvl9\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887061 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887070 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887078 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887087 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887099 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8s4s\" (UniqueName: \"kubernetes.io/projected/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-kube-api-access-z8s4s\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887109 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887117 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887126 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887133 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887142 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887150 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6314229-18f2-4fbf-b206-0e7e36ac4cdd-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887159 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf65fb1-003c-46c0-9f78-18365d2c0d63-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887168 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:03.887298 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:03.887176 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942cb7e8-aba4-46b2-a7bc-31d3443aa31b-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:04.129110 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.129072 5978 generic.go:356] "Generic (PLEG): container finished" podID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerID="1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4" exitCode=0 Oct 31 00:56:04.129300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.129121 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" Oct 31 00:56:04.129300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.129156 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" event={"ID":"cdf65fb1-003c-46c0-9f78-18365d2c0d63","Type":"ContainerDied","Data":"1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4"} Oct 31 00:56:04.129300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.129218 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m" event={"ID":"cdf65fb1-003c-46c0-9f78-18365d2c0d63","Type":"ContainerDied","Data":"23a70f36457e4a97765ceeede21e380d0998c4568927d6f94312e08a4f718250"} Oct 31 00:56:04.129300 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.129247 5978 scope.go:122] "RemoveContainer" containerID="1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4" Oct 31 00:56:04.131149 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.131124 5978 generic.go:356] "Generic (PLEG): container finished" podID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerID="da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463" exitCode=0 Oct 31 00:56:04.131290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.131172 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" Oct 31 00:56:04.131290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.131208 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" event={"ID":"d6314229-18f2-4fbf-b206-0e7e36ac4cdd","Type":"ContainerDied","Data":"da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463"} Oct 31 00:56:04.131290 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.131267 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg" event={"ID":"d6314229-18f2-4fbf-b206-0e7e36ac4cdd","Type":"ContainerDied","Data":"934c80aced48c5c07fc3dcd4d2ec148388396a8b106796335c6e84087073956f"} Oct 31 00:56:04.132917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.132892 5978 generic.go:356] "Generic (PLEG): container finished" podID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerID="c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210" exitCode=0 Oct 31 00:56:04.133046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.132937 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" Oct 31 00:56:04.133046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.132972 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" event={"ID":"942cb7e8-aba4-46b2-a7bc-31d3443aa31b","Type":"ContainerDied","Data":"c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210"} Oct 31 00:56:04.133046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.133023 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz" event={"ID":"942cb7e8-aba4-46b2-a7bc-31d3443aa31b","Type":"ContainerDied","Data":"0cfdacf96fa5cc4ff6c3e95b14c417ee754f64fb852b245b94813a28a909f37e"} Oct 31 00:56:04.144349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.144326 5978 scope.go:122] "RemoveContainer" containerID="1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4" Oct 31 00:56:04.146292 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:56:04.146255 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4\": container with ID starting with 1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4 not found: ID does not exist" containerID="1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4" Oct 31 00:56:04.146407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.146301 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4"} err="failed to get container status \"1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4\": rpc error: code = NotFound desc = could not find container \"1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4\": container with ID starting with 1e91338292414dc5b38fd5e4089e5c1c112e55cdbc264623abe8d5a4428269e4 not found: ID does not exist" Oct 31 00:56:04.146407 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.146328 5978 scope.go:122] "RemoveContainer" containerID="da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463" Oct 31 00:56:04.157557 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.157540 5978 scope.go:122] "RemoveContainer" containerID="6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386" Oct 31 00:56:04.168198 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.168180 5978 scope.go:122] "RemoveContainer" containerID="da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463" Oct 31 00:56:04.168509 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:56:04.168488 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463\": container with ID starting with da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463 not found: ID does not exist" containerID="da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463" Oct 31 00:56:04.168594 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.168518 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463"} err="failed to get container status \"da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463\": rpc error: code = NotFound desc = could not find container \"da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463\": container with ID starting with da6ee8c615c05111532ade3aaab2d9c517e0b0188d9e331e8b3d4ee583281463 not found: ID does not exist" Oct 31 00:56:04.168594 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.168536 5978 scope.go:122] "RemoveContainer" containerID="6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386" Oct 31 00:56:04.168772 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:56:04.168758 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386\": container with ID starting with 6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386 not found: ID does not exist" containerID="6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386" Oct 31 00:56:04.168816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.168777 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386"} err="failed to get container status \"6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386\": rpc error: code = NotFound desc = could not find container \"6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386\": container with ID starting with 6964292f8dcee7408260292b5486653d8ef83c6c2744eb4397fec9364d0be386 not found: ID does not exist" Oct 31 00:56:04.168816 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.168793 5978 scope.go:122] "RemoveContainer" containerID="c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210" Oct 31 00:56:04.179600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.179573 5978 scope.go:122] "RemoveContainer" containerID="c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210" Oct 31 00:56:04.179882 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:56:04.179862 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210\": container with ID starting with c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210 not found: ID does not exist" containerID="c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210" Oct 31 00:56:04.179959 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.179888 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210"} err="failed to get container status \"c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210\": rpc error: code = NotFound desc = could not find container \"c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210\": container with ID starting with c591192ad7951559ce9aa7ba23d8dee6005068aecab5227c7ea4893fb770e210 not found: ID does not exist" Oct 31 00:56:04.209991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.209959 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m"] Oct 31 00:56:04.219218 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.219192 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-6268m"] Oct 31 00:56:04.254944 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.254912 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz"] Oct 31 00:56:04.262824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.262794 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-jt2rz"] Oct 31 00:56:04.293667 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.293579 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg"] Oct 31 00:56:04.301090 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.301060 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-82d887-openshift-default-854d6ffc9c-brrdg"] Oct 31 00:56:04.352184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.352144 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:04.433987 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.433894 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" path="/var/lib/kubelet/pods/942cb7e8-aba4-46b2-a7bc-31d3443aa31b/volumes" Oct 31 00:56:04.434926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.434900 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" path="/var/lib/kubelet/pods/cdf65fb1-003c-46c0-9f78-18365d2c0d63/volumes" Oct 31 00:56:04.435774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:04.435751 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" path="/var/lib/kubelet/pods/d6314229-18f2-4fbf-b206-0e7e36ac4cdd/volumes" Oct 31 00:56:05.150115 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:05.150079 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:05.325917 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:05.325878 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:06.122651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:06.113023 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:07.783316 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:07.783278 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:09.523085 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:09.522947 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:11.194349 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:11.194314 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:12.884166 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:12.884065 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:13.035447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:13.035405 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:14.637596 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:14.637553 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:16.160132 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:16.160080 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:17.744373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:17.742752 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:19.307018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:19.306986 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:20.265080 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:20.265034 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:21.010227 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:21.010188 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:22.600966 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:22.600928 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:22.751764 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:22.751726 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:24.340527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:24.340338 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:24.993031 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:24.992994 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:26.058893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:26.058853 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:26.219547 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:26.214656 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:28.003526 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:28.003489 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:28.240332 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:28.240296 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:29.936214 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:29.936171 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:31.568726 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:31.568677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:32.817527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:32.817447 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:33.190240 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:56:33.190142 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:56:33.273284 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.273244 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:33.577273 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.577237 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:33.704525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.704009 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt"] Oct 31 00:56:33.706168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.706133 5978 cpu_manager.go:414] "containerMap: removing container" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" Oct 31 00:56:33.706340 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.706325 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" Oct 31 00:56:33.706509 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.706496 5978 cpu_manager.go:414] "containerMap: removing container" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" Oct 31 00:56:33.706632 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.706620 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" Oct 31 00:56:33.706749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.706739 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" Oct 31 00:56:33.710946 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.710912 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" Oct 31 00:56:33.711113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.710955 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" Oct 31 00:56:33.711113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.710964 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" Oct 31 00:56:33.711113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.711033 5978 cpu_manager.go:414] "containerMap: removing container" podUID="47678034-2289-4238-8e31-074e2c1e5c78" containerName="payload-processing" Oct 31 00:56:33.711113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.711041 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47678034-2289-4238-8e31-074e2c1e5c78" containerName="payload-processing" Oct 31 00:56:33.711519 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.711443 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="942cb7e8-aba4-46b2-a7bc-31d3443aa31b" containerName="istio-proxy" Oct 31 00:56:33.711519 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.711493 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf65fb1-003c-46c0-9f78-18365d2c0d63" containerName="istio-proxy" Oct 31 00:56:33.711519 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.711509 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" Oct 31 00:56:33.712304 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.712272 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6314229-18f2-4fbf-b206-0e7e36ac4cdd" containerName="istio-proxy" Oct 31 00:56:33.712382 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.712324 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="47678034-2289-4238-8e31-074e2c1e5c78" containerName="payload-processing" Oct 31 00:56:33.736483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.736057 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt"] Oct 31 00:56:33.736483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.736205 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.753472 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.753427 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-82b5e8-openshift-default-dockercfg-tlr4n\"" Oct 31 00:56:33.806754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.806709 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:33.862731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.862622 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.863154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.862768 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-envoy\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.863154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.862853 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-podinfo\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.863154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.862874 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da83d69-b959-4613-9b4d-d869ddb1884c-istiod-ca-cert\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.863154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.862969 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-token\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.863154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.863015 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-data\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.863154 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.863114 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-credential-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.863404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.863172 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-certs\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.863404 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.863263 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcrf\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-kube-api-access-4pcrf\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.963707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.963654 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:33.964623 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.964590 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-token\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.980207 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.980169 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-data\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.980616 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.980512 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-credential-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.981237 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.981211 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-certs\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.981679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.981656 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcrf\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-kube-api-access-4pcrf\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.981926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.981909 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.982104 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.982075 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-envoy\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.982247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.982165 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-podinfo\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.982247 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.982197 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da83d69-b959-4613-9b4d-d869ddb1884c-istiod-ca-cert\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.983311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.983062 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da83d69-b959-4613-9b4d-d869ddb1884c-istiod-ca-cert\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.985206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.983683 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-data\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.985206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.984219 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-credential-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:33.985206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.984440 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-certs\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:34.014477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:33.997146 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-envoy\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:34.014477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:34.007805 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-socket\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:34.018376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:34.018334 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-podinfo\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:34.023950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:34.023914 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-token\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:34.038047 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:34.036113 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcrf\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-kube-api-access-4pcrf\") pod \"e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:34.068479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:34.068015 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:34.461144 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:34.461112 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt"] Oct 31 00:56:34.462753 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:56:34.462725 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da83d69_b959_4613_9b4d_d869ddb1884c.slice/crio-5881f062234050006258de6efb47368c8f6c1314471dfb8a9e16c3eac09a60a0 WatchSource:0}: Error finding container 5881f062234050006258de6efb47368c8f6c1314471dfb8a9e16c3eac09a60a0: Status 404 returned error can't find the container with id 5881f062234050006258de6efb47368c8f6c1314471dfb8a9e16c3eac09a60a0 Oct 31 00:56:34.466047 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:34.466010 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:34.466131 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:34.466097 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:34.466168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:34.466130 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:56:35.360029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:35.359971 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:35.474565 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:35.474522 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" event={"ID":"6da83d69-b959-4613-9b4d-d869ddb1884c","Type":"ContainerStarted","Data":"72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40"} Oct 31 00:56:35.474775 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:35.474582 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" event={"ID":"6da83d69-b959-4613-9b4d-d869ddb1884c","Type":"ContainerStarted","Data":"5881f062234050006258de6efb47368c8f6c1314471dfb8a9e16c3eac09a60a0"} Oct 31 00:56:35.525246 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:35.525164 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" podStartSLOduration=2.525149255 podStartE2EDuration="2.525149255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:35.524551775 +0000 UTC m=+2213.222665542" watchObservedRunningTime="2026-10-31 00:56:35.525149255 +0000 UTC m=+2213.223263019" Oct 31 00:56:36.068374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:36.068343 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:36.075175 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:36.075148 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:36.482677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:36.482600 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:36.483604 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:36.483583 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:56:37.075915 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:37.075863 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:37.215623 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:37.215578 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:38.077474 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:38.077296 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:38.248369 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:38.248319 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:38.431338 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:38.431243 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:39.446484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:39.445736 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:40.692029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:40.690278 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:40.877981 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:40.877837 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:42.408047 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:42.408012 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:44.048923 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:44.048885 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:44.195699 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:44.195655 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:44.347891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:44.347854 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:46.191982 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:46.191946 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:46.815734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:46.815702 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:47.992863 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:47.986725 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:49.685099 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:49.685058 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:50.365100 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:50.365064 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5"] Oct 31 00:56:50.365286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:50.365255 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" containerID="cri-o://da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29" gracePeriod=30 Oct 31 00:56:50.942305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:50.942270 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l"] Oct 31 00:56:50.942807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:50.942550 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" containerID="cri-o://ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c" gracePeriod=30 Oct 31 00:56:51.627603 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:51.626886 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:52.922687 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:52.922646 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:53.099955 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:53.099916 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:53.333083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:53.333047 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:54.605367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:54.605334 5978 patch_prober.go:29] interesting pod/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:56:54.605853 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:54.605403 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:55.180483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.179236 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:55.659945 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.659921 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:56:55.685361 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.685325 5978 generic.go:356] "Generic (PLEG): container finished" podID="c60da948-ebf0-4065-977a-a502ded06209" containerID="da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29" exitCode=0 Oct 31 00:56:55.685569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.685375 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" Oct 31 00:56:55.685569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.685403 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" event={"ID":"c60da948-ebf0-4065-977a-a502ded06209","Type":"ContainerDied","Data":"da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29"} Oct 31 00:56:55.685569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.685466 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5" event={"ID":"c60da948-ebf0-4065-977a-a502ded06209","Type":"ContainerDied","Data":"bf250bd2cc9d5aa880e5af3543e4b195dead8190eae738e393391a8d4757b8bc"} Oct 31 00:56:55.685569 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.685492 5978 scope.go:122] "RemoveContainer" containerID="da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29" Oct 31 00:56:55.698371 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.698347 5978 scope.go:122] "RemoveContainer" containerID="da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29" Oct 31 00:56:55.698686 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:56:55.698666 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29\": container with ID starting with da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29 not found: ID does not exist" containerID="da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29" Oct 31 00:56:55.698736 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.698693 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29"} err="failed to get container status \"da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29\": rpc error: code = NotFound desc = could not find container \"da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29\": container with ID starting with da47cb537ae7262c46acd3ac2e422214a62b758caf542250bdfb9240f25f9e29 not found: ID does not exist" Oct 31 00:56:55.778954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.778870 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60da948-ebf0-4065-977a-a502ded06209-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60da948-ebf0-4065-977a-a502ded06209-istiod-ca-cert\") pod \"c60da948-ebf0-4065-977a-a502ded06209\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " Oct 31 00:56:55.778954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.778921 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-socket\") pod \"c60da948-ebf0-4065-977a-a502ded06209\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " Oct 31 00:56:55.779176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.778966 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-istio-token\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-istio-token\") pod \"c60da948-ebf0-4065-977a-a502ded06209\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " Oct 31 00:56:55.779176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.779069 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-data\") pod \"c60da948-ebf0-4065-977a-a502ded06209\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " Oct 31 00:56:55.779176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.779133 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-certs\") pod \"c60da948-ebf0-4065-977a-a502ded06209\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " Oct 31 00:56:55.779346 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.779266 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c60da948-ebf0-4065-977a-a502ded06209-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60da948-ebf0-4065-977a-a502ded06209-istio-podinfo\") pod \"c60da948-ebf0-4065-977a-a502ded06209\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " Oct 31 00:56:55.779346 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.779317 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-credential-socket\") pod \"c60da948-ebf0-4065-977a-a502ded06209\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " Oct 31 00:56:55.779496 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.779261 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-socket" pod "c60da948-ebf0-4065-977a-a502ded06209" (UID: "c60da948-ebf0-4065-977a-a502ded06209"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:55.779589 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.779567 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-envoy\") pod \"c60da948-ebf0-4065-977a-a502ded06209\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " Oct 31 00:56:55.779730 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.779712 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-kube-api-access-td2mw\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-kube-api-access-td2mw\") pod \"c60da948-ebf0-4065-977a-a502ded06209\" (UID: \"c60da948-ebf0-4065-977a-a502ded06209\") " Oct 31 00:56:55.779804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.779555 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-certs" pod "c60da948-ebf0-4065-977a-a502ded06209" (UID: "c60da948-ebf0-4065-977a-a502ded06209"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:55.779804 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.779768 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-credential-socket" pod "c60da948-ebf0-4065-977a-a502ded06209" (UID: "c60da948-ebf0-4065-977a-a502ded06209"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:55.780579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.780552 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:55.780579 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.780580 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:55.780843 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.780595 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:55.780843 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.780750 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-data" pod "c60da948-ebf0-4065-977a-a502ded06209" (UID: "c60da948-ebf0-4065-977a-a502ded06209"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:55.781040 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.781019 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60da948-ebf0-4065-977a-a502ded06209-istiod-ca-cert" pod "c60da948-ebf0-4065-977a-a502ded06209" (UID: "c60da948-ebf0-4065-977a-a502ded06209"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:55.782152 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.782123 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c60da948-ebf0-4065-977a-a502ded06209-istio-podinfo" pod "c60da948-ebf0-4065-977a-a502ded06209" (UID: "c60da948-ebf0-4065-977a-a502ded06209"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:55.782649 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.782621 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-kube-api-access-td2mw" pod "c60da948-ebf0-4065-977a-a502ded06209" (UID: "c60da948-ebf0-4065-977a-a502ded06209"). InnerVolumeSpecName "kube-api-access-td2mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:55.782926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.782903 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-envoy" pod "c60da948-ebf0-4065-977a-a502ded06209" (UID: "c60da948-ebf0-4065-977a-a502ded06209"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:55.783134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.783117 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-istio-token" pod "c60da948-ebf0-4065-977a-a502ded06209" (UID: "c60da948-ebf0-4065-977a-a502ded06209"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:55.882303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.882267 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60da948-ebf0-4065-977a-a502ded06209-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:55.882303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.882297 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:55.882303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.882307 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td2mw\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-kube-api-access-td2mw\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:55.882601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.882318 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60da948-ebf0-4065-977a-a502ded06209-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:55.882601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.882328 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60da948-ebf0-4065-977a-a502ded06209-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:55.882601 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:55.882335 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60da948-ebf0-4065-977a-a502ded06209-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.061376 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.061351 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5"] Oct 31 00:56:56.069668 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.069631 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-t7kb5"] Oct 31 00:56:56.201606 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.201580 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:56:56.287798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.287761 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-credential-socket\") pod \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " Oct 31 00:56:56.287972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.287879 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istiod-ca-cert\") pod \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " Oct 31 00:56:56.287972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.287923 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-socket\") pod \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " Oct 31 00:56:56.287972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.287944 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-certs\") pod \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " Oct 31 00:56:56.288168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.287993 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-podinfo\") pod \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " Oct 31 00:56:56.288168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.288063 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-kube-api-access-rg4jm\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-kube-api-access-rg4jm\") pod \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " Oct 31 00:56:56.288168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.288113 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-token\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-token\") pod \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " Oct 31 00:56:56.288168 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.288123 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-credential-socket" pod "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" (UID: "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.288377 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.288175 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-data\") pod \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " Oct 31 00:56:56.288377 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.288330 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-envoy\") pod \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\" (UID: \"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1\") " Oct 31 00:56:56.288722 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.288421 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-certs" pod "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" (UID: "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.288722 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.288542 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-socket" pod "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" (UID: "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.288722 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.288553 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istiod-ca-cert" pod "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" (UID: "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:56.288925 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.288701 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-data" pod "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" (UID: "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.289362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.289318 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.289362 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.289350 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.289614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.289370 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.289614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.289383 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.289614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.289397 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.291688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.291661 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-podinfo" pod "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" (UID: "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:56.291688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.291664 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-kube-api-access-rg4jm" pod "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" (UID: "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1"). InnerVolumeSpecName "kube-api-access-rg4jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:56.291976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.291955 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-token" pod "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" (UID: "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:56.292913 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.292891 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-envoy" pod "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" (UID: "3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:56.391234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.391200 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.391234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.391227 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4jm\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-kube-api-access-rg4jm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.391234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.391238 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.391508 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.391248 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:56:56.428234 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.428200 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60da948-ebf0-4065-977a-a502ded06209" path="/var/lib/kubelet/pods/c60da948-ebf0-4065-977a-a502ded06209/volumes" Oct 31 00:56:56.695518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.695391 5978 generic.go:356] "Generic (PLEG): container finished" podID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerID="ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c" exitCode=0 Oct 31 00:56:56.695518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.695442 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" Oct 31 00:56:56.695518 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.695439 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" event={"ID":"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1","Type":"ContainerDied","Data":"ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c"} Oct 31 00:56:56.696109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.695524 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l" event={"ID":"3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1","Type":"ContainerDied","Data":"1857c7d3f870319056876c33a05c47b7f6a4c539ce1244801ae63e502185c84a"} Oct 31 00:56:56.696109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.695549 5978 scope.go:122] "RemoveContainer" containerID="ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c" Oct 31 00:56:56.707280 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.707253 5978 scope.go:122] "RemoveContainer" containerID="ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c" Oct 31 00:56:56.707775 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:56:56.707751 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c\": container with ID starting with ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c not found: ID does not exist" containerID="ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c" Oct 31 00:56:56.707856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.707781 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c"} err="failed to get container status \"ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c\": rpc error: code = NotFound desc = could not find container \"ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c\": container with ID starting with ef69efd1fdf3c42a662928660a5d50c64729d2df61b3a2d20718f8061afee20c not found: ID does not exist" Oct 31 00:56:56.746589 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.746554 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l"] Oct 31 00:56:56.769855 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.769823 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-xtn2l"] Oct 31 00:56:56.822307 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:56.822274 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:58.408789 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:58.408742 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:58.452501 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:56:58.450187 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" path="/var/lib/kubelet/pods/3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1/volumes" Oct 31 00:57:00.005711 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:00.005416 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:00.381374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:00.381335 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:01.687580 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:01.687529 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:03.144000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:03.143958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:03.183504 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:57:03.183468 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:57:03.874340 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:03.874303 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:04.002774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:04.002742 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk"] Oct 31 00:57:04.008997 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:04.008969 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bdb61f-tgzlk"] Oct 31 00:57:04.034137 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:04.034100 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:04.182089 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:04.182008 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:04.427984 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:04.427944 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27bcad0-d9a7-4346-9c09-8f34040c9fc7" path="/var/lib/kubelet/pods/d27bcad0-d9a7-4346-9c09-8f34040c9fc7/volumes" Oct 31 00:57:04.564942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:04.564860 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:05.965322 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:05.965288 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:07.763088 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:07.763049 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:08.296887 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:08.296853 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:09.423725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:09.423692 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:11.080741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:11.080658 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:11.384377 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:11.384296 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:11.548011 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:11.547975 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:11.776499 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:11.776376 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq"] Oct 31 00:57:11.777096 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:11.777055 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" containerID="cri-o://6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c" gracePeriod=30 Oct 31 00:57:11.788350 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:11.788312 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:12.737739 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:12.737699 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:13.354206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.354167 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2"] Oct 31 00:57:13.355068 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.355048 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" Oct 31 00:57:13.355068 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.355068 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" Oct 31 00:57:13.355201 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.355096 5978 cpu_manager.go:414] "containerMap: removing container" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" Oct 31 00:57:13.355201 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.355102 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" Oct 31 00:57:13.355302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.355282 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60da948-ebf0-4065-977a-a502ded06209" containerName="istio-proxy" Oct 31 00:57:13.355302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.355294 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3dd9a6-e180-471c-8d4c-74dbfc8e1dc1" containerName="istio-proxy" Oct 31 00:57:13.359174 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.359144 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.364787 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.364754 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:13.379316 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.379283 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c942c9-openshift-default-dockercfg-7s9hc\"" Oct 31 00:57:13.395195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.395158 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2"] Oct 31 00:57:13.462930 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.462890 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-envoy\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.463121 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.462948 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-data\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.463121 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.463054 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-certs\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.463279 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.463177 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-podinfo\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.463279 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.463205 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istiod-ca-cert\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.463644 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.463615 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnrl\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-kube-api-access-flnrl\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.463774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.463672 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-credential-socket\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.463774 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.463731 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-token\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.463869 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.463773 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-socket\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.565477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.565124 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnrl\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-kube-api-access-flnrl\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.565477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.565186 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-credential-socket\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.565477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.565235 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-token\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.565477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.565276 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-socket\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.565477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.565394 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-envoy\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.565932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.565577 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-data\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.565932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.565659 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-certs\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.565932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.565789 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-podinfo\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.565932 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.565816 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istiod-ca-cert\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.574478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.566921 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istiod-ca-cert\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.578477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.575714 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-credential-socket\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.578477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.576472 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-socket\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.613082 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.593345 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-certs\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.613082 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.593680 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-data\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.622381 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.621510 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:13.635337 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.634545 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnrl\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-kube-api-access-flnrl\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.635866 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.635843 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-envoy\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.645528 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.645083 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-token\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.679255 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.679177 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-podinfo\") pod \"e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:13.977268 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:13.977186 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:14.301766 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:57:14.301733 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c26f4af_e8d0_4ad9_95f7_46d841598ea0.slice/crio-2c6c2c71ed354d069f72e5044cc9a9edbf87404c3228bb1b8b85f41068514ab5 WatchSource:0}: Error finding container 2c6c2c71ed354d069f72e5044cc9a9edbf87404c3228bb1b8b85f41068514ab5: Status 404 returned error can't find the container with id 2c6c2c71ed354d069f72e5044cc9a9edbf87404c3228bb1b8b85f41068514ab5 Oct 31 00:57:14.303060 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:14.303030 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2"] Oct 31 00:57:14.305023 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:14.304985 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:57:14.305153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:14.305118 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:57:14.305153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:14.305151 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:57:15.174002 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:15.173967 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" event={"ID":"5c26f4af-e8d0-4ad9-95f7-46d841598ea0","Type":"ContainerStarted","Data":"92bca105e6f7526d9a0c2115e34c2e70200172004fb9f3568c18b9f264d05e98"} Oct 31 00:57:15.174432 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:15.174025 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" event={"ID":"5c26f4af-e8d0-4ad9-95f7-46d841598ea0","Type":"ContainerStarted","Data":"2c6c2c71ed354d069f72e5044cc9a9edbf87404c3228bb1b8b85f41068514ab5"} Oct 31 00:57:15.230561 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:15.230504 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" podStartSLOduration=2.230476737 podStartE2EDuration="2.230476737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:15.226863153 +0000 UTC m=+2252.924976919" watchObservedRunningTime="2026-10-31 00:57:15.230476737 +0000 UTC m=+2252.928590503" Oct 31 00:57:15.286840 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:15.286794 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:15.977920 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:15.977883 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:15.984715 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:15.984688 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:16.186945 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:16.186905 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:16.191677 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:16.191645 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:57:17.082017 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.081989 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:57:17.146662 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.146624 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f3833c-293b-41e4-878a-a36a42207ff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f3833c-293b-41e4-878a-a36a42207ff2-istiod-ca-cert\") pod \"36f3833c-293b-41e4-878a-a36a42207ff2\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " Oct 31 00:57:17.146879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.146759 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-socket\") pod \"36f3833c-293b-41e4-878a-a36a42207ff2\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " Oct 31 00:57:17.146879 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.146831 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-certs\") pod \"36f3833c-293b-41e4-878a-a36a42207ff2\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " Oct 31 00:57:17.147054 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.146976 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-credential-socket\") pod \"36f3833c-293b-41e4-878a-a36a42207ff2\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " Oct 31 00:57:17.147109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147043 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f3833c-293b-41e4-878a-a36a42207ff2-istiod-ca-cert" pod "36f3833c-293b-41e4-878a-a36a42207ff2" (UID: "36f3833c-293b-41e4-878a-a36a42207ff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:17.147339 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147315 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-socket" pod "36f3833c-293b-41e4-878a-a36a42207ff2" (UID: "36f3833c-293b-41e4-878a-a36a42207ff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:17.147483 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147445 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-kube-api-access-wwmnn\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-kube-api-access-wwmnn\") pod \"36f3833c-293b-41e4-878a-a36a42207ff2\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " Oct 31 00:57:17.147558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147507 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-data\") pod \"36f3833c-293b-41e4-878a-a36a42207ff2\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " Oct 31 00:57:17.147558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147518 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-certs" pod "36f3833c-293b-41e4-878a-a36a42207ff2" (UID: "36f3833c-293b-41e4-878a-a36a42207ff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:17.147558 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147550 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-envoy\") pod \"36f3833c-293b-41e4-878a-a36a42207ff2\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " Oct 31 00:57:17.147688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147642 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36f3833c-293b-41e4-878a-a36a42207ff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f3833c-293b-41e4-878a-a36a42207ff2-istio-podinfo\") pod \"36f3833c-293b-41e4-878a-a36a42207ff2\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " Oct 31 00:57:17.147737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147689 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-credential-socket" pod "36f3833c-293b-41e4-878a-a36a42207ff2" (UID: "36f3833c-293b-41e4-878a-a36a42207ff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:17.147737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147727 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-istio-token\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-istio-token\") pod \"36f3833c-293b-41e4-878a-a36a42207ff2\" (UID: \"36f3833c-293b-41e4-878a-a36a42207ff2\") " Oct 31 00:57:17.147863 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.147844 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-data" pod "36f3833c-293b-41e4-878a-a36a42207ff2" (UID: "36f3833c-293b-41e4-878a-a36a42207ff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:17.148511 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.148486 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:57:17.148511 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.148512 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:57:17.148707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.148526 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f3833c-293b-41e4-878a-a36a42207ff2-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:57:17.148707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.148541 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:57:17.148707 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.148553 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:57:17.150848 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.150275 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-kube-api-access-wwmnn" pod "36f3833c-293b-41e4-878a-a36a42207ff2" (UID: "36f3833c-293b-41e4-878a-a36a42207ff2"). InnerVolumeSpecName "kube-api-access-wwmnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:17.150969 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.150904 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-envoy" pod "36f3833c-293b-41e4-878a-a36a42207ff2" (UID: "36f3833c-293b-41e4-878a-a36a42207ff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:17.151265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.151237 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36f3833c-293b-41e4-878a-a36a42207ff2-istio-podinfo" pod "36f3833c-293b-41e4-878a-a36a42207ff2" (UID: "36f3833c-293b-41e4-878a-a36a42207ff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:57:17.151369 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.151352 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-istio-token" pod "36f3833c-293b-41e4-878a-a36a42207ff2" (UID: "36f3833c-293b-41e4-878a-a36a42207ff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:17.198352 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.198315 5978 generic.go:356] "Generic (PLEG): container finished" podID="36f3833c-293b-41e4-878a-a36a42207ff2" containerID="6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c" exitCode=0 Oct 31 00:57:17.198820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.198377 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" Oct 31 00:57:17.198820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.198392 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" event={"ID":"36f3833c-293b-41e4-878a-a36a42207ff2","Type":"ContainerDied","Data":"6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c"} Oct 31 00:57:17.198820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.198446 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq" event={"ID":"36f3833c-293b-41e4-878a-a36a42207ff2","Type":"ContainerDied","Data":"dd533d6f308162f01d0482ceb558c0576406103191b4c52f8c0d015164cf9c4f"} Oct 31 00:57:17.198820 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.198491 5978 scope.go:122] "RemoveContainer" containerID="6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c" Oct 31 00:57:17.211390 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.211364 5978 scope.go:122] "RemoveContainer" containerID="9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823" Oct 31 00:57:17.222416 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.222392 5978 scope.go:122] "RemoveContainer" containerID="6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c" Oct 31 00:57:17.222861 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:57:17.222828 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c\": container with ID starting with 6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c not found: ID does not exist" containerID="6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c" Oct 31 00:57:17.222991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.222867 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c"} err="failed to get container status \"6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c\": rpc error: code = NotFound desc = could not find container \"6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c\": container with ID starting with 6139ab1f4c8935e0cf35eb3d396c8cdec0561762dba9bf0e6d19ba498487015c not found: ID does not exist" Oct 31 00:57:17.222991 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.222888 5978 scope.go:122] "RemoveContainer" containerID="9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823" Oct 31 00:57:17.223218 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:57:17.223192 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823\": container with ID starting with 9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823 not found: ID does not exist" containerID="9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823" Oct 31 00:57:17.223305 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.223225 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823"} err="failed to get container status \"9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823\": rpc error: code = NotFound desc = could not find container \"9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823\": container with ID starting with 9b447a1c7f58f8c7a29c74964f42a0ba5869563c59d708600138ec27d1ee3823 not found: ID does not exist" Oct 31 00:57:17.259078 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.258900 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:17.259586 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.259318 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq"] Oct 31 00:57:17.263498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.260989 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwmnn\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-kube-api-access-wwmnn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:57:17.263498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.261022 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f3833c-293b-41e4-878a-a36a42207ff2-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:57:17.263498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.261041 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f3833c-293b-41e4-878a-a36a42207ff2-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:57:17.263498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.261053 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f3833c-293b-41e4-878a-a36a42207ff2-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:57:17.268512 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.266844 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bdb61f-openshift-default-75c44c9f85-pq6sq"] Oct 31 00:57:17.849502 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:17.849441 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:18.053819 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:18.053779 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:18.427891 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:18.427859 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" path="/var/lib/kubelet/pods/36f3833c-293b-41e4-878a-a36a42207ff2/volumes" Oct 31 00:57:19.579342 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.579303 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:19.737291 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.737247 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:19.755053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.755018 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk"] Oct 31 00:57:19.756178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.756157 5978 cpu_manager.go:414] "containerMap: removing container" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" Oct 31 00:57:19.756254 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.756182 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" Oct 31 00:57:19.756254 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.756211 5978 cpu_manager.go:414] "containerMap: removing container" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" Oct 31 00:57:19.756254 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.756219 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" Oct 31 00:57:19.756442 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.756430 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" Oct 31 00:57:19.756503 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.756464 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f3833c-293b-41e4-878a-a36a42207ff2" containerName="istio-proxy" Oct 31 00:57:19.762291 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.762265 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:19.780255 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.780222 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk"] Oct 31 00:57:19.784281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.784246 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c942c9\"" Oct 31 00:57:19.784281 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.784256 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c942c9-dockercfg-fxzrq\"" Oct 31 00:57:19.855570 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.851116 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k"] Oct 31 00:57:19.857253 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.857222 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:19.896661 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.888603 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k"] Oct 31 00:57:19.911791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.910952 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kch\" (UniqueName: \"kubernetes.io/projected/23d8fc09-02c3-4eff-a125-4b7d4966ef49-kube-api-access-f9kch\") pod \"payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk\" (UID: \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:19.911791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.911305 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d8fc09-02c3-4eff-a125-4b7d4966ef49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk\" (UID: \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:19.911791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.911355 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czl95\" (UniqueName: \"kubernetes.io/projected/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-kube-api-access-czl95\") pod \"payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k\" (UID: \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:19.911791 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:19.911437 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k\" (UID: \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:20.069544 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.069442 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d8fc09-02c3-4eff-a125-4b7d4966ef49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk\" (UID: \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:20.069753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.069555 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czl95\" (UniqueName: \"kubernetes.io/projected/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-kube-api-access-czl95\") pod \"payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k\" (UID: \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:20.069753 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.069654 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k\" (UID: \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:20.069893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.069778 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kch\" (UniqueName: \"kubernetes.io/projected/23d8fc09-02c3-4eff-a125-4b7d4966ef49-kube-api-access-f9kch\") pod \"payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk\" (UID: \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:20.074230 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.074147 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k\" (UID: \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:20.074469 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.074416 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d8fc09-02c3-4eff-a125-4b7d4966ef49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk\" (UID: \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:20.102800 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.102760 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czl95\" (UniqueName: \"kubernetes.io/projected/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-kube-api-access-czl95\") pod \"payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k\" (UID: \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:20.103653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.103623 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kch\" (UniqueName: \"kubernetes.io/projected/23d8fc09-02c3-4eff-a125-4b7d4966ef49-kube-api-access-f9kch\") pod \"payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk\" (UID: \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:20.115954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.115856 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:20.377873 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.377781 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:20.438618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.437938 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k"] Oct 31 00:57:20.443523 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:57:20.442586 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e08c87e_bbc1_4fbd_8881_e00ebf387a78.slice/crio-4b8d3b4bea65711176a5dc27a3dcc9be3967ca41a87d2f3b21d32fa8c4e8027f WatchSource:0}: Error finding container 4b8d3b4bea65711176a5dc27a3dcc9be3967ca41a87d2f3b21d32fa8c4e8027f: Status 404 returned error can't find the container with id 4b8d3b4bea65711176a5dc27a3dcc9be3967ca41a87d2f3b21d32fa8c4e8027f Oct 31 00:57:20.685961 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:20.685926 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk"] Oct 31 00:57:20.686715 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:57:20.686691 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d8fc09_02c3_4eff_a125_4b7d4966ef49.slice/crio-eb46a5ca5bc81a9e1c976377f10e13aa9907810bb3c95768066ee11100832cd0 WatchSource:0}: Error finding container eb46a5ca5bc81a9e1c976377f10e13aa9907810bb3c95768066ee11100832cd0: Status 404 returned error can't find the container with id eb46a5ca5bc81a9e1c976377f10e13aa9907810bb3c95768066ee11100832cd0 Oct 31 00:57:21.250508 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:21.250463 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" event={"ID":"1e08c87e-bbc1-4fbd-8881-e00ebf387a78","Type":"ContainerStarted","Data":"17b00932f135066f4c85fd19344ed4461ec691aaa9bc502b3873565f907edcbe"} Oct 31 00:57:21.250771 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:21.250542 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" event={"ID":"1e08c87e-bbc1-4fbd-8881-e00ebf387a78","Type":"ContainerStarted","Data":"4b8d3b4bea65711176a5dc27a3dcc9be3967ca41a87d2f3b21d32fa8c4e8027f"} Oct 31 00:57:21.250771 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:21.250577 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:21.252283 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:21.252256 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" event={"ID":"23d8fc09-02c3-4eff-a125-4b7d4966ef49","Type":"ContainerStarted","Data":"f900426fb48e2c2962a183f4eb36db310325dacf9333bb9e44182eeb9118078c"} Oct 31 00:57:21.252388 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:21.252297 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" event={"ID":"23d8fc09-02c3-4eff-a125-4b7d4966ef49","Type":"ContainerStarted","Data":"eb46a5ca5bc81a9e1c976377f10e13aa9907810bb3c95768066ee11100832cd0"} Oct 31 00:57:21.252431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:21.252405 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:21.290238 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:21.290178 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" podStartSLOduration=2.29016077 podStartE2EDuration="2.29016077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:21.286738341 +0000 UTC m=+2258.984852107" watchObservedRunningTime="2026-10-31 00:57:21.29016077 +0000 UTC m=+2258.988274534" Oct 31 00:57:21.327275 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:21.327207 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" podStartSLOduration=2.327191869 podStartE2EDuration="2.327191869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:21.32224081 +0000 UTC m=+2259.020354577" watchObservedRunningTime="2026-10-31 00:57:21.327191869 +0000 UTC m=+2259.025305633" Oct 31 00:57:21.581291 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:21.581259 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:23.409954 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:23.409903 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:23.553714 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:23.553682 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:23.731256 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:23.731173 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:25.084239 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:25.082811 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:26.820490 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:26.806937 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:27.060853 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:27.060818 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:27.266286 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:27.266199 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:28.508918 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:28.508880 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:30.374186 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:30.374144 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:32.266743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:32.266208 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:32.268053 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:32.268025 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 00:57:32.268765 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:32.268747 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 00:57:33.185094 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:57:33.185056 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:57:34.116423 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:34.116391 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:35.711422 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:35.711383 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:36.096947 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:36.096911 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:36.724872 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:36.724835 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:37.208220 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.208178 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t"] Oct 31 00:57:37.217576 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.217544 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.246650 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.246620 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4f4e69-openshift-default-dockercfg-lj2kc\"" Oct 31 00:57:37.252319 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.252280 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t"] Oct 31 00:57:37.293129 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.293080 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-certs\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.294024 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.293984 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-data\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.294195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.294044 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr8wm\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-kube-api-access-wr8wm\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.294195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.294138 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-credential-socket\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.294320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.294192 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-podinfo\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.294320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.294246 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istiod-ca-cert\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.294320 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.294301 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-envoy\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.294614 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.294587 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-token\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.294744 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.294678 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-socket\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.316766 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.316724 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.404723 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-certs\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.404823 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-data\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.404861 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8wm\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-kube-api-access-wr8wm\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.404895 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-credential-socket\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.404925 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-podinfo\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.404976 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istiod-ca-cert\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.405008 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-envoy\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.405137 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-token\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.405210 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-socket\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.405676 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-certs\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.405756 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.405686 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-socket\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.406522 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.405900 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-credential-socket\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.445505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.423659 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istiod-ca-cert\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.445505 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.434821 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-data\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.521565 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.512814 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:37.558475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.545823 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-envoy\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.567475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.565929 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-token\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.567475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.567380 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-podinfo\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.592623 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.592288 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8wm\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-kube-api-access-wr8wm\") pod \"e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:37.848460 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:37.848405 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:38.181565 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:38.181532 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t"] Oct 31 00:57:38.182219 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:57:38.182193 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bde62e5_02bf_4b4b_bfd7_4022e8616ab8.slice/crio-b2e030545c514f9bac043d0a7c51158fbad81e9895c2d280d271de4776eaa90d WatchSource:0}: Error finding container b2e030545c514f9bac043d0a7c51158fbad81e9895c2d280d271de4776eaa90d: Status 404 returned error can't find the container with id b2e030545c514f9bac043d0a7c51158fbad81e9895c2d280d271de4776eaa90d Oct 31 00:57:38.185437 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:38.185400 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:57:38.185583 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:38.185502 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:57:38.185583 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:38.185554 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:57:38.533373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:38.533258 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" event={"ID":"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8","Type":"ContainerStarted","Data":"bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5"} Oct 31 00:57:38.533373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:38.533344 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" event={"ID":"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8","Type":"ContainerStarted","Data":"b2e030545c514f9bac043d0a7c51158fbad81e9895c2d280d271de4776eaa90d"} Oct 31 00:57:38.574632 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:38.574579 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" podStartSLOduration=1.5745636410000001 podStartE2EDuration="1.574563641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:38.573623196 +0000 UTC m=+2276.271736963" watchObservedRunningTime="2026-10-31 00:57:38.574563641 +0000 UTC m=+2276.272677396" Oct 31 00:57:38.848652 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:38.848610 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:39.332176 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:39.328324 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:39.855373 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:39.855334 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:40.550037 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:40.550006 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:40.551185 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:40.551164 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:57:41.056386 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:41.056350 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:41.218970 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:41.218934 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:42.806016 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:42.805979 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:42.962607 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:42.962558 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:43.333278 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:43.333237 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:45.001093 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:45.001002 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:46.516554 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:46.516522 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:46.943393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:46.943358 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:48.228265 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:48.228225 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:49.893098 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:49.893057 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:51.416685 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:51.416640 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:53.008836 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:53.008796 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:54.614010 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:54.613968 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:56.265881 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:56.265833 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:57.814335 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:57.814295 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:59.020962 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:59.020924 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:59.175815 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:59.175765 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:59.434807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:57:59.434704 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:01.352702 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:01.352622 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:01.512466 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:01.512407 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:03.190129 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:58:03.190088 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:58:04.151478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:04.150561 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:06.022772 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:06.022736 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:08.005897 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:08.005843 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:09.717942 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:09.717904 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:11.313228 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:11.313190 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:13.305596 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:13.305468 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:13.906700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:13.906668 5978 generic.go:356] "Generic (PLEG): container finished" podID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerID="72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40" exitCode=0 Oct 31 00:58:13.906700 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:13.906707 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" event={"ID":"6da83d69-b959-4613-9b4d-d869ddb1884c","Type":"ContainerDied","Data":"72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40"} Oct 31 00:58:13.907184 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:13.907169 5978 scope.go:122] "RemoveContainer" containerID="72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40" Oct 31 00:58:13.909256 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:13.909220 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:58:13.909384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:13.909316 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:58:13.909384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:13.909371 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:58:14.499145 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:14.499111 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:14.693512 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:14.693478 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:14.945182 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:14.945146 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" event={"ID":"6da83d69-b959-4613-9b4d-d869ddb1884c","Type":"ContainerStarted","Data":"521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3"} Oct 31 00:58:15.040436 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:15.040399 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:15.072628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:15.071541 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:58:15.072628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:15.072111 5978 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:58:15.072628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:15.072188 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 00:58:15.792006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:15.791970 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:16.069095 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:16.069058 5978 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:58:16.069303 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:16.069115 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 00:58:16.907895 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:16.907856 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:17.068564 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:17.068521 5978 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:58:17.068779 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:17.068592 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 00:58:17.337737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:17.337040 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:18.069204 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:18.069167 5978 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:58:18.069706 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:18.069224 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 00:58:18.778833 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:18.778792 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:19.069203 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:19.069163 5978 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:58:19.069417 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:19.069236 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 00:58:20.068773 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:20.068732 5978 patch_prober.go:29] interesting pod/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 00:58:20.069006 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:20.068812 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 00:58:20.641106 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:20.640776 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:21.074950 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:21.074914 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:58:21.075447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:21.075430 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:58:21.076334 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:21.076315 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 00:58:22.293311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:22.293268 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:24.023577 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:24.023534 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:25.704430 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:25.704394 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:27.222447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.222413 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27"] Oct 31 00:58:27.226365 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.226344 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.242798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.242766 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4f4e69-serving-cert\"" Oct 31 00:58:27.257678 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.257642 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27"] Oct 31 00:58:27.344754 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.344701 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:27.387921 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.387871 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.388126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.387964 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rc2\" (UniqueName: \"kubernetes.io/projected/ea368ff9-fb95-48f5-99eb-c73749668dd6-kube-api-access-k4rc2\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.388126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.388000 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-metrics-certs\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.388126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.388095 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea368ff9-fb95-48f5-99eb-c73749668dd6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.490016 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.489926 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.490016 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.490004 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rc2\" (UniqueName: \"kubernetes.io/projected/ea368ff9-fb95-48f5-99eb-c73749668dd6-kube-api-access-k4rc2\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.490410 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.490043 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-metrics-certs\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.490410 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:58:27.490084 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-serving-cert: secret "maas-api-e2e-inf-b-4f4e69-serving-cert" not found Oct 31 00:58:27.490410 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:58:27.490142 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls podName:ea368ff9-fb95-48f5-99eb-c73749668dd6 nodeName:}" failed. No retries permitted until 2026-10-31 00:58:27.990125958 +0000 UTC m=+2325.688239721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls") pod "maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" (UID: "ea368ff9-fb95-48f5-99eb-c73749668dd6") : secret "maas-api-e2e-inf-b-4f4e69-serving-cert" not found Oct 31 00:58:27.490410 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.490158 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea368ff9-fb95-48f5-99eb-c73749668dd6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.490970 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.490944 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:27.491654 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.491628 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea368ff9-fb95-48f5-99eb-c73749668dd6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.493752 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.493732 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-metrics-certs\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.513393 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.513345 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rc2\" (UniqueName: \"kubernetes.io/projected/ea368ff9-fb95-48f5-99eb-c73749668dd6-kube-api-access-k4rc2\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:27.723516 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.723478 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:27.999224 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:27.999191 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:28.001828 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:28.001801 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls\") pod \"maas-api-e2e-inf-b-4f4e69-5456985799-mcf27\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:28.145223 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:28.145193 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:28.479413 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:28.479382 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27"] Oct 31 00:58:28.481267 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:58:28.481244 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea368ff9_fb95_48f5_99eb_c73749668dd6.slice/crio-f26e4a32c764894119a2d0b6eae79a1c2683846e44b29d1e8cf3aeb0010edab9 WatchSource:0}: Error finding container f26e4a32c764894119a2d0b6eae79a1c2683846e44b29d1e8cf3aeb0010edab9: Status 404 returned error can't find the container with id f26e4a32c764894119a2d0b6eae79a1c2683846e44b29d1e8cf3aeb0010edab9 Oct 31 00:58:28.482743 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:28.482724 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:58:29.079642 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:29.079597 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:29.117069 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:29.117024 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" event={"ID":"ea368ff9-fb95-48f5-99eb-c73749668dd6","Type":"ContainerStarted","Data":"5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d"} Oct 31 00:58:29.117302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:29.117148 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" event={"ID":"ea368ff9-fb95-48f5-99eb-c73749668dd6","Type":"ContainerStarted","Data":"f26e4a32c764894119a2d0b6eae79a1c2683846e44b29d1e8cf3aeb0010edab9"} Oct 31 00:58:29.117302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:29.117209 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:29.161240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:29.161181 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" podStartSLOduration=1.8370336680000001 podStartE2EDuration="2.161163145s" totalImagesPullingTime="324.129477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:29.158524515 +0000 UTC m=+2326.856638332" watchObservedRunningTime="2026-10-31 00:58:29.161163145 +0000 UTC m=+2326.859276909" Oct 31 00:58:30.827605 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:30.823818 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:32.417125 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:32.417090 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:33.183506 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:58:33.183468 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:58:34.219999 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:34.219959 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:35.130858 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:35.130828 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 00:58:35.809311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:35.809272 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p"] Oct 31 00:58:35.809828 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:35.809503 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" containerID="cri-o://efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525" gracePeriod=30 Oct 31 00:58:36.001976 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:36.001934 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:37.740484 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:37.726561 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:39.311514 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:39.303701 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:40.934776 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:40.934663 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:41.162598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.162566 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:58:41.221639 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.221598 5978 generic.go:356] "Generic (PLEG): container finished" podID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerID="efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525" exitCode=0 Oct 31 00:58:41.222798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.221699 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" event={"ID":"570b0220-fcbf-41bf-8437-9b18ee3d81af","Type":"ContainerDied","Data":"efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525"} Oct 31 00:58:41.222798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.221715 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" Oct 31 00:58:41.222798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.221804 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p" event={"ID":"570b0220-fcbf-41bf-8437-9b18ee3d81af","Type":"ContainerDied","Data":"f1278e2ba44919326114fd71948c285f5da9785766b01d37746b8fbe52db5c6a"} Oct 31 00:58:41.222798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.221844 5978 scope.go:122] "RemoveContainer" containerID="efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525" Oct 31 00:58:41.236465 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.236423 5978 scope.go:122] "RemoveContainer" containerID="efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525" Oct 31 00:58:41.236902 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:58:41.236871 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525\": container with ID starting with efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525 not found: ID does not exist" containerID="efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525" Oct 31 00:58:41.237103 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.237063 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525"} err="failed to get container status \"efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525\": rpc error: code = NotFound desc = could not find container \"efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525\": container with ID starting with efcbec024398df4c59228fbf7a5befcb8f434d77d2387764712c255225571525 not found: ID does not exist" Oct 31 00:58:41.277391 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.277338 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-token\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-token\") pod \"570b0220-fcbf-41bf-8437-9b18ee3d81af\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " Oct 31 00:58:41.277625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.277425 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-envoy\") pod \"570b0220-fcbf-41bf-8437-9b18ee3d81af\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " Oct 31 00:58:41.277625 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.277564 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570b0220-fcbf-41bf-8437-9b18ee3d81af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b0220-fcbf-41bf-8437-9b18ee3d81af-istiod-ca-cert\") pod \"570b0220-fcbf-41bf-8437-9b18ee3d81af\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " Oct 31 00:58:41.277764 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.277674 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-certs\") pod \"570b0220-fcbf-41bf-8437-9b18ee3d81af\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " Oct 31 00:58:41.277998 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.277973 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-certs" pod "570b0220-fcbf-41bf-8437-9b18ee3d81af" (UID: "570b0220-fcbf-41bf-8437-9b18ee3d81af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:41.278143 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.278042 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570b0220-fcbf-41bf-8437-9b18ee3d81af-istiod-ca-cert" pod "570b0220-fcbf-41bf-8437-9b18ee3d81af" (UID: "570b0220-fcbf-41bf-8437-9b18ee3d81af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:41.278219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.278140 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-credential-socket\") pod \"570b0220-fcbf-41bf-8437-9b18ee3d81af\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " Oct 31 00:58:41.278219 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.278173 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-socket\") pod \"570b0220-fcbf-41bf-8437-9b18ee3d81af\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " Oct 31 00:58:41.278575 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.278226 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-podinfo\") pod \"570b0220-fcbf-41bf-8437-9b18ee3d81af\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " Oct 31 00:58:41.278575 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.278328 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-kube-api-access-2wktv\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-kube-api-access-2wktv\") pod \"570b0220-fcbf-41bf-8437-9b18ee3d81af\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " Oct 31 00:58:41.278782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.278679 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-data\") pod \"570b0220-fcbf-41bf-8437-9b18ee3d81af\" (UID: \"570b0220-fcbf-41bf-8437-9b18ee3d81af\") " Oct 31 00:58:41.279173 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.279123 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-credential-socket" pod "570b0220-fcbf-41bf-8437-9b18ee3d81af" (UID: "570b0220-fcbf-41bf-8437-9b18ee3d81af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:41.279719 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.279573 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-socket" pod "570b0220-fcbf-41bf-8437-9b18ee3d81af" (UID: "570b0220-fcbf-41bf-8437-9b18ee3d81af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:41.279719 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.279682 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-data" pod "570b0220-fcbf-41bf-8437-9b18ee3d81af" (UID: "570b0220-fcbf-41bf-8437-9b18ee3d81af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:41.280141 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.280124 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:58:41.280272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.280146 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:58:41.280272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.280161 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:58:41.280272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.280176 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570b0220-fcbf-41bf-8437-9b18ee3d81af-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:58:41.280272 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.280188 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:58:41.280935 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.280910 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-token" pod "570b0220-fcbf-41bf-8437-9b18ee3d81af" (UID: "570b0220-fcbf-41bf-8437-9b18ee3d81af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:41.281177 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.281153 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-envoy" pod "570b0220-fcbf-41bf-8437-9b18ee3d81af" (UID: "570b0220-fcbf-41bf-8437-9b18ee3d81af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:41.281723 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.281699 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-podinfo" pod "570b0220-fcbf-41bf-8437-9b18ee3d81af" (UID: "570b0220-fcbf-41bf-8437-9b18ee3d81af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:58:41.282087 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.282066 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-kube-api-access-2wktv" pod "570b0220-fcbf-41bf-8437-9b18ee3d81af" (UID: "570b0220-fcbf-41bf-8437-9b18ee3d81af"). InnerVolumeSpecName "kube-api-access-2wktv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:41.381710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.381613 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wktv\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-kube-api-access-2wktv\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:58:41.381710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.381645 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:58:41.381710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.381656 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:58:41.381710 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.381665 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570b0220-fcbf-41bf-8437-9b18ee3d81af-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:58:41.570501 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.570468 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p"] Oct 31 00:58:41.579737 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:41.579699 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4k69p"] Oct 31 00:58:42.428711 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:42.428677 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" path="/var/lib/kubelet/pods/570b0220-fcbf-41bf-8437-9b18ee3d81af/volumes" Oct 31 00:58:42.819083 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:42.819023 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:44.353948 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:44.353908 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:46.115367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:46.115325 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:47.094134 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:47.094100 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:47.278197 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:47.278154 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:47.797244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:47.793392 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:49.486488 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:49.477181 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:51.373653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:51.370374 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:52.618600 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:52.618549 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:53.177582 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:53.177536 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:54.918537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:54.917659 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:55.524195 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:55.524152 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:56.130635 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.130597 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:56.547538 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.547397 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt"] Oct 31 00:58:56.548372 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.548346 5978 cpu_manager.go:414] "containerMap: removing container" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" Oct 31 00:58:56.548372 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.548370 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" Oct 31 00:58:56.548692 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.548675 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="570b0220-fcbf-41bf-8437-9b18ee3d81af" containerName="istio-proxy" Oct 31 00:58:56.554984 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.554954 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.577208 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.577178 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6f8856-openshift-default-dockercfg-k84qx\"" Oct 31 00:58:56.608922 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.597190 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt"] Oct 31 00:58:56.673478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.664870 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-certs\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.715227 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.673873 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.770656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.770611 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-podinfo\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.776968 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.776933 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istiod-ca-cert\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.806817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.806711 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-data\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.815741 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.815697 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-envoy\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.849762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.848905 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-token\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.849762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.848966 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d944j\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-kube-api-access-d944j\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.849762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.849036 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-credential-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:56.849762 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:56.838014 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.015795 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-certs\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.015871 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.015945 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-podinfo\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.015982 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istiod-ca-cert\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.016047 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-data\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.016191 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-envoy\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.016366 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-token\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.016441 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d944j\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-kube-api-access-d944j\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.016542 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-credential-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.017360 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-credential-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.019890 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-data\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.021478 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.020188 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-socket\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.075534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.049249 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-certs\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.075534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.050213 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:57.075534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.051060 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istiod-ca-cert\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.075534 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.070183 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-envoy\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.142479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.138591 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d944j\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-kube-api-access-d944j\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.142479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.140474 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-podinfo\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.174967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.170042 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-token\") pod \"e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.174967 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.174779 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:57.664959 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.664931 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt"] Oct 31 00:58:57.666440 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:58:57.666414 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3068a265_cfd5_41b1_b2ed_ddac0d8553ab.slice/crio-949d05e3434cc74ece77d0ec4d62b54ef2efb9d795cae7ecac2bf3b348d57b8b WatchSource:0}: Error finding container 949d05e3434cc74ece77d0ec4d62b54ef2efb9d795cae7ecac2bf3b348d57b8b: Status 404 returned error can't find the container with id 949d05e3434cc74ece77d0ec4d62b54ef2efb9d795cae7ecac2bf3b348d57b8b Oct 31 00:58:57.669851 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.669811 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:58:57.669973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.669901 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:58:57.669973 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:57.669949 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:58:58.397941 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:58.397899 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" event={"ID":"3068a265-cfd5-41b1-b2ed-ddac0d8553ab","Type":"ContainerStarted","Data":"1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d"} Oct 31 00:58:58.398447 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:58.397974 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" event={"ID":"3068a265-cfd5-41b1-b2ed-ddac0d8553ab","Type":"ContainerStarted","Data":"949d05e3434cc74ece77d0ec4d62b54ef2efb9d795cae7ecac2bf3b348d57b8b"} Oct 31 00:58:58.475311 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:58.475249 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" podStartSLOduration=2.475234179 podStartE2EDuration="2.475234179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:58.462251878 +0000 UTC m=+2356.160365647" watchObservedRunningTime="2026-10-31 00:58:58.475234179 +0000 UTC m=+2356.173348017" Oct 31 00:58:58.795370 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:58.795250 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:59.176364 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:59.176328 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:59.182824 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:59.182796 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:59.408610 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:59.408578 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:58:59.409809 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:58:59.409790 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 00:59:00.554392 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:00.553914 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:00.965525 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:00.958224 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:01.150479 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:01.150173 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:02.551539 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:02.531285 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:02.750084 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:02.750049 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:02.939064 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:02.938981 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:03.349743 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:59:03.349695 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:59:04.317346 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:04.317315 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:06.021653 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:06.021592 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:07.673029 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:07.672988 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:09.353358 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:09.352630 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:10.567267 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:10.567234 5978 generic.go:356] "Generic (PLEG): container finished" podID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerID="92bca105e6f7526d9a0c2115e34c2e70200172004fb9f3568c18b9f264d05e98" exitCode=0 Oct 31 00:59:10.567985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:10.567318 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" event={"ID":"5c26f4af-e8d0-4ad9-95f7-46d841598ea0","Type":"ContainerDied","Data":"92bca105e6f7526d9a0c2115e34c2e70200172004fb9f3568c18b9f264d05e98"} Oct 31 00:59:10.567985 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:10.567904 5978 scope.go:122] "RemoveContainer" containerID="92bca105e6f7526d9a0c2115e34c2e70200172004fb9f3568c18b9f264d05e98" Oct 31 00:59:10.570046 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:10.570009 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:59:10.570166 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:10.570083 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:59:10.570166 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:10.570116 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:59:11.016354 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:11.016268 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:11.577632 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:11.577593 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" event={"ID":"5c26f4af-e8d0-4ad9-95f7-46d841598ea0","Type":"ContainerStarted","Data":"e227f25fb86fb607f3c8531274dbbd2d771d5fd41fe617b8bb3f61309b314f36"} Oct 31 00:59:11.592612 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:11.592574 5978 patch_prober.go:29] interesting pod/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:59:11.592798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:11.592643 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:59:11.978278 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:11.978179 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:59:11.980183 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:11.980152 5978 patch_prober.go:29] interesting pod/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:59:11.980333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:11.980226 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 00:59:12.588178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:12.588138 5978 generic.go:356] "Generic (PLEG): container finished" podID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerID="bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5" exitCode=0 Oct 31 00:59:12.588178 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:12.588143 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" event={"ID":"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8","Type":"ContainerDied","Data":"bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5"} Oct 31 00:59:12.588825 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:12.588613 5978 scope.go:122] "RemoveContainer" containerID="bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5" Oct 31 00:59:12.590499 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:12.590464 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:59:12.590618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:12.590541 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:59:12.590618 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:12.590590 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:59:12.717000 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:12.715758 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:12.978628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:12.978524 5978 patch_prober.go:29] interesting pod/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:59:12.978628 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:12.978603 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 00:59:13.599100 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:13.599064 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" event={"ID":"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8","Type":"ContainerStarted","Data":"d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917"} Oct 31 00:59:13.857113 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:13.857017 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:59:13.857309 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:13.857111 5978 patch_prober.go:29] interesting pod/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:59:13.857309 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:13.857163 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 00:59:13.978461 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:13.978408 5978 patch_prober.go:29] interesting pod/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:59:13.978679 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:13.978486 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 00:59:14.441206 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:14.436964 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:14.852989 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:14.852946 5978 patch_prober.go:29] interesting pod/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:59:14.853651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:14.853616 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 00:59:14.978220 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:14.978180 5978 patch_prober.go:29] interesting pod/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:59:14.978405 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:14.978238 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 00:59:15.848972 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:15.848926 5978 patch_prober.go:29] interesting pod/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:59:15.849292 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:15.849262 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 00:59:15.984475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:15.983944 5978 patch_prober.go:29] interesting pod/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:59:15.984475 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:15.984015 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 00:59:16.190553 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:16.190411 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:59:16.448798 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:16.444845 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:16.853476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:16.853035 5978 patch_prober.go:29] interesting pod/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:59:16.853476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:16.853099 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 00:59:16.978109 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:16.978068 5978 patch_prober.go:29] interesting pod/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:59:16.978302 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:16.978127 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 00:59:17.848841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:17.848798 5978 patch_prober.go:29] interesting pod/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:59:17.849293 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:17.848854 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 00:59:17.983537 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:17.983486 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:59:17.985333 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:17.985308 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 00:59:18.144656 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:18.143946 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:18.601926 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:18.601882 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:18.848893 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:18.848845 5978 patch_prober.go:29] interesting pod/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:59:18.849374 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:18.848919 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 00:59:19.986476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:19.980251 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:19.986476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:19.984108 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:59:19.986476 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:19.984884 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:59:19.994507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:19.994480 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 00:59:20.985527 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:20.985448 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:21.667511 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:21.666519 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:22.728807 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:22.728770 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx"] Oct 31 00:59:22.732856 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:22.732829 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:22.748604 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:22.748577 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:22.758094 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:22.758056 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:59:22.764035 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:22.764000 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx"] Oct 31 00:59:22.931287 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:22.931233 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f157460d-jh8hx\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:22.931498 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:22.931474 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w84z\" (UniqueName: \"kubernetes.io/projected/2610e058-1053-48c7-a656-abe44618f8f6-kube-api-access-8w84z\") pod \"maas-api-revoke-sub-f157460d-jh8hx\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:22.931623 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:22.931607 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f157460d-jh8hx\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:23.032868 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.032758 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f157460d-jh8hx\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:23.033059 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.032953 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w84z\" (UniqueName: \"kubernetes.io/projected/2610e058-1053-48c7-a656-abe44618f8f6-kube-api-access-8w84z\") pod \"maas-api-revoke-sub-f157460d-jh8hx\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:23.033128 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.033069 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f157460d-jh8hx\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:23.033715 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.033691 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f157460d-jh8hx\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:23.035310 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.035283 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f157460d-jh8hx\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:23.054202 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.054166 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w84z\" (UniqueName: \"kubernetes.io/projected/2610e058-1053-48c7-a656-abe44618f8f6-kube-api-access-8w84z\") pod \"maas-api-revoke-sub-f157460d-jh8hx\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:23.348126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.348084 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:23.373734 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.373691 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:23.705795 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.705761 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx"] Oct 31 00:59:23.707064 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:59:23.707040 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2610e058_1053_48c7_a656_abe44618f8f6.slice/crio-5be27240ab6ea4247c44ef5fcab96b68bffdad7606724c69040fa7d891d02b65 WatchSource:0}: Error finding container 5be27240ab6ea4247c44ef5fcab96b68bffdad7606724c69040fa7d891d02b65: Status 404 returned error can't find the container with id 5be27240ab6ea4247c44ef5fcab96b68bffdad7606724c69040fa7d891d02b65 Oct 31 00:59:23.741769 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:23.741733 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" event={"ID":"2610e058-1053-48c7-a656-abe44618f8f6","Type":"ContainerStarted","Data":"5be27240ab6ea4247c44ef5fcab96b68bffdad7606724c69040fa7d891d02b65"} Oct 31 00:59:24.752126 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:24.752091 5978 generic.go:356] "Generic (PLEG): container finished" podID="2610e058-1053-48c7-a656-abe44618f8f6" containerID="98c811f1b6a1a5e188fac637d5357cd81e4689e47424f9ff4de5f81e79ea0a5f" exitCode=0 Oct 31 00:59:24.752563 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:24.752175 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" event={"ID":"2610e058-1053-48c7-a656-abe44618f8f6","Type":"ContainerDied","Data":"98c811f1b6a1a5e188fac637d5357cd81e4689e47424f9ff4de5f81e79ea0a5f"} Oct 31 00:59:24.999199 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:24.999078 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:25.933193 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:25.933162 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:25.981940 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:25.981907 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-openshift-ca\") pod \"2610e058-1053-48c7-a656-abe44618f8f6\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " Oct 31 00:59:25.982153 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:25.982131 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2610e058-1053-48c7-a656-abe44618f8f6-kube-api-access-8w84z\" (UniqueName: \"kubernetes.io/projected/2610e058-1053-48c7-a656-abe44618f8f6-kube-api-access-8w84z\") pod \"2610e058-1053-48c7-a656-abe44618f8f6\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " Oct 31 00:59:25.982250 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:25.982195 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-xks-ca\") pod \"2610e058-1053-48c7-a656-abe44618f8f6\" (UID: \"2610e058-1053-48c7-a656-abe44618f8f6\") " Oct 31 00:59:25.982315 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:25.982300 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-openshift-ca" pod "2610e058-1053-48c7-a656-abe44618f8f6" (UID: "2610e058-1053-48c7-a656-abe44618f8f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:25.983244 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:25.983217 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:59:25.984749 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:25.984722 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-xks-ca" pod "2610e058-1053-48c7-a656-abe44618f8f6" (UID: "2610e058-1053-48c7-a656-abe44618f8f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:25.985651 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:25.985615 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2610e058-1053-48c7-a656-abe44618f8f6-kube-api-access-8w84z" pod "2610e058-1053-48c7-a656-abe44618f8f6" (UID: "2610e058-1053-48c7-a656-abe44618f8f6"). InnerVolumeSpecName "kube-api-access-8w84z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:26.085072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:26.085039 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w84z\" (UniqueName: \"kubernetes.io/projected/2610e058-1053-48c7-a656-abe44618f8f6-kube-api-access-8w84z\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:59:26.085072 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:26.085069 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2610e058-1053-48c7-a656-abe44618f8f6-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:59:26.619911 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:26.619872 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:26.770052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:26.769972 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" Oct 31 00:59:26.770052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:26.770002 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx" event={"ID":"2610e058-1053-48c7-a656-abe44618f8f6","Type":"ContainerDied","Data":"5be27240ab6ea4247c44ef5fcab96b68bffdad7606724c69040fa7d891d02b65"} Oct 31 00:59:26.770052 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:26.770055 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be27240ab6ea4247c44ef5fcab96b68bffdad7606724c69040fa7d891d02b65" Oct 31 00:59:28.281085 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:28.281051 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:29.927358 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:29.927319 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:31.531384 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:31.531339 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:33.183417 ip-10-0-129-199 kubenswrapper[5978]: E1031 00:59:33.183371 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 00:59:33.208241 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:33.205756 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:34.879325 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:34.879287 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:36.507883 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:36.507847 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:38.223410 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:38.223338 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:38.393367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:38.393328 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:39.815720 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:39.815679 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:41.386012 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:41.385902 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:43.030264 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:43.030227 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:44.505716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:44.505680 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:44.666477 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:44.662430 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:45.069254 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.069216 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5"] Oct 31 00:59:45.070102 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.070084 5978 cpu_manager.go:414] "containerMap: removing container" podUID="2610e058-1053-48c7-a656-abe44618f8f6" containerName="revoke-keys" Oct 31 00:59:45.070161 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.070105 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2610e058-1053-48c7-a656-abe44618f8f6" containerName="revoke-keys" Oct 31 00:59:45.070240 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.070231 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="2610e058-1053-48c7-a656-abe44618f8f6" containerName="revoke-keys" Oct 31 00:59:45.074521 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.074489 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.098367 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.098331 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:45.098782 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.098756 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 00:59:45.104731 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.104703 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5"] Oct 31 00:59:45.223270 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.223230 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt97x\" (UniqueName: \"kubernetes.io/projected/e0675706-0707-4335-a6a7-3c73d90329cf-kube-api-access-pt97x\") pod \"maas-api-revoke-sub-54b022a6-k9hm5\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.223507 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.223487 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b022a6-k9hm5\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.223641 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.223620 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b022a6-k9hm5\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.325446 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.325359 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt97x\" (UniqueName: \"kubernetes.io/projected/e0675706-0707-4335-a6a7-3c73d90329cf-kube-api-access-pt97x\") pod \"maas-api-revoke-sub-54b022a6-k9hm5\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.325640 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.325562 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b022a6-k9hm5\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.325640 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.325633 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b022a6-k9hm5\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.326209 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.326182 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b022a6-k9hm5\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.328034 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.328012 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b022a6-k9hm5\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.345852 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.345824 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt97x\" (UniqueName: \"kubernetes.io/projected/e0675706-0707-4335-a6a7-3c73d90329cf-kube-api-access-pt97x\") pod \"maas-api-revoke-sub-54b022a6-k9hm5\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.397598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.397569 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:45.703716 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.703679 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5"] Oct 31 00:59:45.704366 ip-10-0-129-199 kubenswrapper[5978]: W1031 00:59:45.704329 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0675706_0707_4335_a6a7_3c73d90329cf.slice/crio-b6fd8d1f80a177fe0128113e5d8e684e60fa028545a5b9f311de4365f42486e5 WatchSource:0}: Error finding container b6fd8d1f80a177fe0128113e5d8e684e60fa028545a5b9f311de4365f42486e5: Status 404 returned error can't find the container with id b6fd8d1f80a177fe0128113e5d8e684e60fa028545a5b9f311de4365f42486e5 Oct 31 00:59:45.938688 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:45.938653 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" event={"ID":"e0675706-0707-4335-a6a7-3c73d90329cf","Type":"ContainerStarted","Data":"b6fd8d1f80a177fe0128113e5d8e684e60fa028545a5b9f311de4365f42486e5"} Oct 31 00:59:46.475132 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:46.459483 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:46.601796 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:46.601760 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:46.751598 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:46.751497 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:47.087151 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:47.087107 5978 generic.go:356] "Generic (PLEG): container finished" podID="e0675706-0707-4335-a6a7-3c73d90329cf" containerID="f3561bd54f2c2e06ce3d5c7b23fcaeb7a91d69c50e64239f2ca6216d85449352" exitCode=0 Oct 31 00:59:47.087597 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:47.087565 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" event={"ID":"e0675706-0707-4335-a6a7-3c73d90329cf","Type":"ContainerDied","Data":"f3561bd54f2c2e06ce3d5c7b23fcaeb7a91d69c50e64239f2ca6216d85449352"} Oct 31 00:59:47.441264 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:47.440844 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:48.302257 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.302229 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:48.393898 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.393797 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-openshift-ca\") pod \"e0675706-0707-4335-a6a7-3c73d90329cf\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " Oct 31 00:59:48.394093 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.393970 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0675706-0707-4335-a6a7-3c73d90329cf-kube-api-access-pt97x\" (UniqueName: \"kubernetes.io/projected/e0675706-0707-4335-a6a7-3c73d90329cf-kube-api-access-pt97x\") pod \"e0675706-0707-4335-a6a7-3c73d90329cf\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " Oct 31 00:59:48.394160 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.394131 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-xks-ca\") pod \"e0675706-0707-4335-a6a7-3c73d90329cf\" (UID: \"e0675706-0707-4335-a6a7-3c73d90329cf\") " Oct 31 00:59:48.394897 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.394865 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-openshift-ca" pod "e0675706-0707-4335-a6a7-3c73d90329cf" (UID: "e0675706-0707-4335-a6a7-3c73d90329cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:48.397018 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.396978 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-xks-ca" pod "e0675706-0707-4335-a6a7-3c73d90329cf" (UID: "e0675706-0707-4335-a6a7-3c73d90329cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:48.397274 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.397249 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0675706-0707-4335-a6a7-3c73d90329cf-kube-api-access-pt97x" pod "e0675706-0707-4335-a6a7-3c73d90329cf" (UID: "e0675706-0707-4335-a6a7-3c73d90329cf"). InnerVolumeSpecName "kube-api-access-pt97x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:48.496678 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.496636 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:59:48.496678 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.496677 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt97x\" (UniqueName: \"kubernetes.io/projected/e0675706-0707-4335-a6a7-3c73d90329cf-kube-api-access-pt97x\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:59:48.497001 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.496692 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0675706-0707-4335-a6a7-3c73d90329cf-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 00:59:48.500784 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:48.500750 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:49.107841 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:49.107809 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" Oct 31 00:59:49.108022 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:49.107812 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5" event={"ID":"e0675706-0707-4335-a6a7-3c73d90329cf","Type":"ContainerDied","Data":"b6fd8d1f80a177fe0128113e5d8e684e60fa028545a5b9f311de4365f42486e5"} Oct 31 00:59:49.108022 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:49.107938 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6fd8d1f80a177fe0128113e5d8e684e60fa028545a5b9f311de4365f42486e5" Oct 31 00:59:50.306658 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:50.306618 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:50.424294 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:50.424267 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:59:50.424689 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:50.424671 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 00:59:51.996041 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:51.996005 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:52.540369 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:52.540337 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:52.691817 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:52.691781 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:53.630725 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:53.630607 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:55.306465 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:55.306384 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:56.809566 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:56.809536 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:57.031957 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:57.031907 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:58.520577 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:58.520530 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:58.658431 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:58.658398 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:58.817883 ip-10-0-129-199 kubenswrapper[5978]: I1031 00:59:58.817848 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:00.172675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.172632 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr"] Oct 31 01:00:00.174071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.174044 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e0675706-0707-4335-a6a7-3c73d90329cf" containerName="revoke-keys" Oct 31 01:00:00.174071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.174071 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0675706-0707-4335-a6a7-3c73d90329cf" containerName="revoke-keys" Oct 31 01:00:00.174274 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.174261 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0675706-0707-4335-a6a7-3c73d90329cf" containerName="revoke-keys" Oct 31 01:00:00.179510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.179465 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f"] Oct 31 01:00:00.179676 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.179582 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:00.185664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.185635 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h"] Oct 31 01:00:00.185800 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.185769 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:00.190586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.190563 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:00.206678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.206646 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr"] Oct 31 01:00:00.207792 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.207771 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:00:00.244846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.244813 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f"] Oct 31 01:00:00.250726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.250673 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc"] Oct 31 01:00:00.256993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.256963 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:00.257424 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.257401 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h"] Oct 31 01:00:00.273338 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.273301 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx"] Oct 31 01:00:00.281634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.281553 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:00.287499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.284578 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc"] Oct 31 01:00:00.287499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.287208 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pg4c\" (UniqueName: \"kubernetes.io/projected/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-kube-api-access-7pg4c\") pod \"maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f\" (UID: \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:00.287499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.287255 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f\" (UID: \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:00.287499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.287291 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c884e066-0089-4a02-a980-6bcf9b5c8e89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h\" (UID: \"c884e066-0089-4a02-a980-6bcf9b5c8e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:00.287499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.287409 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8wv\" (UniqueName: \"kubernetes.io/projected/f1a039bc-2f7e-4626-afef-28baa7d75b44-kube-api-access-xp8wv\") pod \"maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr\" (UID: \"f1a039bc-2f7e-4626-afef-28baa7d75b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:00.287892 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.287617 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca573556-53d4-431a-8f75-1637ebaa618b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc\" (UID: \"ca573556-53d4-431a-8f75-1637ebaa618b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:00.287892 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.287663 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wmc\" (UniqueName: \"kubernetes.io/projected/c884e066-0089-4a02-a980-6bcf9b5c8e89-kube-api-access-q8wmc\") pod \"maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h\" (UID: \"c884e066-0089-4a02-a980-6bcf9b5c8e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:00.287892 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.287710 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a039bc-2f7e-4626-afef-28baa7d75b44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr\" (UID: \"f1a039bc-2f7e-4626-afef-28baa7d75b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:00.287892 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.287753 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6ddp\" (UniqueName: \"kubernetes.io/projected/ca573556-53d4-431a-8f75-1637ebaa618b-kube-api-access-j6ddp\") pod \"maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc\" (UID: \"ca573556-53d4-431a-8f75-1637ebaa618b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:00.289917 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.289896 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr"] Oct 31 01:00:00.302919 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.302672 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:00.324139 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.324103 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx"] Oct 31 01:00:00.345516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.344336 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr"] Oct 31 01:00:00.350266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.350217 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf"] Oct 31 01:00:00.358567 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.357406 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q"] Oct 31 01:00:00.363071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.363038 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:00.363234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.363187 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf"] Oct 31 01:00:00.363298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.363273 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:00.388324 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.388295 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw"] Oct 31 01:00:00.389575 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.389545 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ed1af6-7b4d-459d-86d6-799672dd7ded-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr\" (UID: \"69ed1af6-7b4d-459d-86d6-799672dd7ded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:00.389699 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.389624 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf\" (UID: \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:00.389764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.389731 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mg8c\" (UniqueName: \"kubernetes.io/projected/69ed1af6-7b4d-459d-86d6-799672dd7ded-kube-api-access-7mg8c\") pod \"maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr\" (UID: \"69ed1af6-7b4d-459d-86d6-799672dd7ded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:00.389802 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.389786 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca573556-53d4-431a-8f75-1637ebaa618b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc\" (UID: \"ca573556-53d4-431a-8f75-1637ebaa618b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:00.389854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.389832 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wmc\" (UniqueName: \"kubernetes.io/projected/c884e066-0089-4a02-a980-6bcf9b5c8e89-kube-api-access-q8wmc\") pod \"maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h\" (UID: \"c884e066-0089-4a02-a980-6bcf9b5c8e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.391553 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a039bc-2f7e-4626-afef-28baa7d75b44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr\" (UID: \"f1a039bc-2f7e-4626-afef-28baa7d75b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.391610 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6ddp\" (UniqueName: \"kubernetes.io/projected/ca573556-53d4-431a-8f75-1637ebaa618b-kube-api-access-j6ddp\") pod \"maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc\" (UID: \"ca573556-53d4-431a-8f75-1637ebaa618b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.391645 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcf361c-5cc7-483e-93f2-063c1e38e0df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q\" (UID: \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.391775 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pg4c\" (UniqueName: \"kubernetes.io/projected/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-kube-api-access-7pg4c\") pod \"maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f\" (UID: \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.391811 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f\" (UID: \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.391841 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c884e066-0089-4a02-a980-6bcf9b5c8e89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h\" (UID: \"c884e066-0089-4a02-a980-6bcf9b5c8e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.391882 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cmlc\" (UniqueName: \"kubernetes.io/projected/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-kube-api-access-6cmlc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf\" (UID: \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.391944 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g24gr\" (UniqueName: \"kubernetes.io/projected/380d9685-5509-41ec-85ea-f4488a8e1461-kube-api-access-g24gr\") pod \"maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx\" (UID: \"380d9685-5509-41ec-85ea-f4488a8e1461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.392033 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8wv\" (UniqueName: \"kubernetes.io/projected/f1a039bc-2f7e-4626-afef-28baa7d75b44-kube-api-access-xp8wv\") pod \"maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr\" (UID: \"f1a039bc-2f7e-4626-afef-28baa7d75b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.392081 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxvt\" (UniqueName: \"kubernetes.io/projected/4bcf361c-5cc7-483e-93f2-063c1e38e0df-kube-api-access-pzxvt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q\" (UID: \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.392111 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d9685-5509-41ec-85ea-f4488a8e1461-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx\" (UID: \"380d9685-5509-41ec-85ea-f4488a8e1461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.392370 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a039bc-2f7e-4626-afef-28baa7d75b44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr\" (UID: \"f1a039bc-2f7e-4626-afef-28baa7d75b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:00.393053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.390870 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca573556-53d4-431a-8f75-1637ebaa618b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc\" (UID: \"ca573556-53d4-431a-8f75-1637ebaa618b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:00.394064 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.393185 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c884e066-0089-4a02-a980-6bcf9b5c8e89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h\" (UID: \"c884e066-0089-4a02-a980-6bcf9b5c8e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:00.394064 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.393349 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f\" (UID: \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:00.396563 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.396544 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:00.413139 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.412669 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q"] Oct 31 01:00:00.436507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.436365 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8wv\" (UniqueName: \"kubernetes.io/projected/f1a039bc-2f7e-4626-afef-28baa7d75b44-kube-api-access-xp8wv\") pod \"maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr\" (UID: \"f1a039bc-2f7e-4626-afef-28baa7d75b44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:00.441983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.441008 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw"] Oct 31 01:00:00.441983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.441061 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj"] Oct 31 01:00:00.442898 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.442853 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wmc\" (UniqueName: \"kubernetes.io/projected/c884e066-0089-4a02-a980-6bcf9b5c8e89-kube-api-access-q8wmc\") pod \"maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h\" (UID: \"c884e066-0089-4a02-a980-6bcf9b5c8e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:00.448518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.445750 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6ddp\" (UniqueName: \"kubernetes.io/projected/ca573556-53d4-431a-8f75-1637ebaa618b-kube-api-access-j6ddp\") pod \"maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc\" (UID: \"ca573556-53d4-431a-8f75-1637ebaa618b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:00.448518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.448210 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pg4c\" (UniqueName: \"kubernetes.io/projected/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-kube-api-access-7pg4c\") pod \"maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f\" (UID: \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:00.459501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.453384 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:00.499816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.498281 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:00.516755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.513440 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g24gr\" (UniqueName: \"kubernetes.io/projected/380d9685-5509-41ec-85ea-f4488a8e1461-kube-api-access-g24gr\") pod \"maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx\" (UID: \"380d9685-5509-41ec-85ea-f4488a8e1461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:00.516755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.513584 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxvt\" (UniqueName: \"kubernetes.io/projected/4bcf361c-5cc7-483e-93f2-063c1e38e0df-kube-api-access-pzxvt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q\" (UID: \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:00.516755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.513618 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d9685-5509-41ec-85ea-f4488a8e1461-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx\" (UID: \"380d9685-5509-41ec-85ea-f4488a8e1461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:00.516755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.513721 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ed1af6-7b4d-459d-86d6-799672dd7ded-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr\" (UID: \"69ed1af6-7b4d-459d-86d6-799672dd7ded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:00.516755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.513757 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf\" (UID: \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:00.516755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.513854 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mg8c\" (UniqueName: \"kubernetes.io/projected/69ed1af6-7b4d-459d-86d6-799672dd7ded-kube-api-access-7mg8c\") pod \"maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr\" (UID: \"69ed1af6-7b4d-459d-86d6-799672dd7ded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:00.516755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.514011 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcf361c-5cc7-483e-93f2-063c1e38e0df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q\" (UID: \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:00.516755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.514181 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cmlc\" (UniqueName: \"kubernetes.io/projected/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-kube-api-access-6cmlc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf\" (UID: \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:00.520550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.517867 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ed1af6-7b4d-459d-86d6-799672dd7ded-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr\" (UID: \"69ed1af6-7b4d-459d-86d6-799672dd7ded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:00.521131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.521070 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d9685-5509-41ec-85ea-f4488a8e1461-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx\" (UID: \"380d9685-5509-41ec-85ea-f4488a8e1461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:00.541959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.525252 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcf361c-5cc7-483e-93f2-063c1e38e0df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q\" (UID: \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:00.541959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.525313 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:00.541959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.531697 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf\" (UID: \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:00.541959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.533276 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj"] Oct 31 01:00:00.541959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.533332 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb"] Oct 31 01:00:00.541959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.538826 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:00.541959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.540308 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:00.564319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.559958 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:00.564319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.562098 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g24gr\" (UniqueName: \"kubernetes.io/projected/380d9685-5509-41ec-85ea-f4488a8e1461-kube-api-access-g24gr\") pod \"maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx\" (UID: \"380d9685-5509-41ec-85ea-f4488a8e1461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:00.564319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.564071 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxvt\" (UniqueName: \"kubernetes.io/projected/4bcf361c-5cc7-483e-93f2-063c1e38e0df-kube-api-access-pzxvt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q\" (UID: \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:00.564579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.564432 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cmlc\" (UniqueName: \"kubernetes.io/projected/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-kube-api-access-6cmlc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf\" (UID: \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:00.566304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.566276 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb"] Oct 31 01:00:00.570912 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.570858 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mg8c\" (UniqueName: \"kubernetes.io/projected/69ed1af6-7b4d-459d-86d6-799672dd7ded-kube-api-access-7mg8c\") pod \"maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr\" (UID: \"69ed1af6-7b4d-459d-86d6-799672dd7ded\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:00.593289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.593242 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:00.618439 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.618398 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxqjz\" (UniqueName: \"kubernetes.io/projected/d3d3f24c-70f4-4199-b581-d4190e64eb61-kube-api-access-rxqjz\") pod \"maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj\" (UID: \"d3d3f24c-70f4-4199-b581-d4190e64eb61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:00.618637 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.618547 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw\" (UID: \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:00.618863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.618725 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlvvn\" (UniqueName: \"kubernetes.io/projected/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-kube-api-access-tlvvn\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw\" (UID: \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:00.618967 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.618952 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d3f24c-70f4-4199-b581-d4190e64eb61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj\" (UID: \"d3d3f24c-70f4-4199-b581-d4190e64eb61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:00.629035 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.628775 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:00.638829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.638802 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:00.692656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.691891 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:00.710852 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.710681 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:00.724977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.720739 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlvvn\" (UniqueName: \"kubernetes.io/projected/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-kube-api-access-tlvvn\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw\" (UID: \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:00.724977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.720814 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f0be5-c787-4cb0-841d-96a71464d09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb\" (UID: \"449f0be5-c787-4cb0-841d-96a71464d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:00.724977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.720960 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwbp\" (UniqueName: \"kubernetes.io/projected/449f0be5-c787-4cb0-841d-96a71464d09a-kube-api-access-sjwbp\") pod \"maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb\" (UID: \"449f0be5-c787-4cb0-841d-96a71464d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:00.724977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.721109 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d3f24c-70f4-4199-b581-d4190e64eb61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj\" (UID: \"d3d3f24c-70f4-4199-b581-d4190e64eb61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:00.724977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.721292 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxqjz\" (UniqueName: \"kubernetes.io/projected/d3d3f24c-70f4-4199-b581-d4190e64eb61-kube-api-access-rxqjz\") pod \"maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj\" (UID: \"d3d3f24c-70f4-4199-b581-d4190e64eb61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:00.724977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.721374 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw\" (UID: \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:00.724977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.721767 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d3f24c-70f4-4199-b581-d4190e64eb61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj\" (UID: \"d3d3f24c-70f4-4199-b581-d4190e64eb61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:00.724977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.722198 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw\" (UID: \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:00.749025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.748959 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxqjz\" (UniqueName: \"kubernetes.io/projected/d3d3f24c-70f4-4199-b581-d4190e64eb61-kube-api-access-rxqjz\") pod \"maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj\" (UID: \"d3d3f24c-70f4-4199-b581-d4190e64eb61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:00.757061 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.756995 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlvvn\" (UniqueName: \"kubernetes.io/projected/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-kube-api-access-tlvvn\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw\" (UID: \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:00.825505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.823822 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f0be5-c787-4cb0-841d-96a71464d09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb\" (UID: \"449f0be5-c787-4cb0-841d-96a71464d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:00.825505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.823923 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwbp\" (UniqueName: \"kubernetes.io/projected/449f0be5-c787-4cb0-841d-96a71464d09a-kube-api-access-sjwbp\") pod \"maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb\" (UID: \"449f0be5-c787-4cb0-841d-96a71464d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:00.825505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.825032 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f0be5-c787-4cb0-841d-96a71464d09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb\" (UID: \"449f0be5-c787-4cb0-841d-96a71464d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:00.850316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.850286 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:00.851356 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.851299 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwbp\" (UniqueName: \"kubernetes.io/projected/449f0be5-c787-4cb0-841d-96a71464d09a-kube-api-access-sjwbp\") pod \"maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb\" (UID: \"449f0be5-c787-4cb0-841d-96a71464d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:00.908423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.908334 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr"] Oct 31 01:00:00.908723 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:00:00.908682 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a039bc_2f7e_4626_afef_28baa7d75b44.slice/crio-33d5472c6845a555b3a9e9a82d253d855f279744b28aba37736e3ece5303cc3e WatchSource:0}: Error finding container 33d5472c6845a555b3a9e9a82d253d855f279744b28aba37736e3ece5303cc3e: Status 404 returned error can't find the container with id 33d5472c6845a555b3a9e9a82d253d855f279744b28aba37736e3ece5303cc3e Oct 31 01:00:00.942905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.942826 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:00.965585 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:00.965504 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:01.123162 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.123126 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f"] Oct 31 01:00:01.132903 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:00:01.132864 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca573556_53d4_431a_8f75_1637ebaa618b.slice/crio-488a6da16a21c5ebb102f6697569b58adcd51ddda9db4281dbc9116e18d913e6 WatchSource:0}: Error finding container 488a6da16a21c5ebb102f6697569b58adcd51ddda9db4281dbc9116e18d913e6: Status 404 returned error can't find the container with id 488a6da16a21c5ebb102f6697569b58adcd51ddda9db4281dbc9116e18d913e6 Oct 31 01:00:01.134748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.134717 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc"] Oct 31 01:00:01.171121 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.171086 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h"] Oct 31 01:00:01.208601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.208569 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw"] Oct 31 01:00:01.212538 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:00:01.212512 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4b1f4c_2e52_43de_b9cd_b7496cbfa3c9.slice/crio-dfeadb4031c0d8fbf6e352902cfb583f2dc6d16b5b831ababf58ffd8164adcd2 WatchSource:0}: Error finding container dfeadb4031c0d8fbf6e352902cfb583f2dc6d16b5b831ababf58ffd8164adcd2: Status 404 returned error can't find the container with id dfeadb4031c0d8fbf6e352902cfb583f2dc6d16b5b831ababf58ffd8164adcd2 Oct 31 01:00:01.240781 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.239859 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" event={"ID":"c884e066-0089-4a02-a980-6bcf9b5c8e89","Type":"ContainerStarted","Data":"1d2ef467748c741d2b2c046accddbe08cd63e8f10ea5b174a7a35514057227b0"} Oct 31 01:00:01.245617 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.245579 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" event={"ID":"f1a039bc-2f7e-4626-afef-28baa7d75b44","Type":"ContainerStarted","Data":"33d5472c6845a555b3a9e9a82d253d855f279744b28aba37736e3ece5303cc3e"} Oct 31 01:00:01.247404 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.247376 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" event={"ID":"b7a3049f-1270-4d3c-a8ad-f13ffe40a214","Type":"ContainerStarted","Data":"9b66f76e46b774610f1c6e043bb3cae6e6d2a58a3d46157d8ddf5e763a446331"} Oct 31 01:00:01.247558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.247435 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx"] Oct 31 01:00:01.248866 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.248837 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" event={"ID":"ca573556-53d4-431a-8f75-1637ebaa618b","Type":"ContainerStarted","Data":"488a6da16a21c5ebb102f6697569b58adcd51ddda9db4281dbc9116e18d913e6"} Oct 31 01:00:01.260729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.260701 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr"] Oct 31 01:00:01.290531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.290494 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf"] Oct 31 01:00:01.299640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.299550 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q"] Oct 31 01:00:01.304537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.304511 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj"] Oct 31 01:00:01.319043 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:00:01.319011 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380d9685_5509_41ec_85ea_f4488a8e1461.slice/crio-9327e594c266f7dc98c242856f0a43c486d0f232b0b16b3e1fb1f19daa416297 WatchSource:0}: Error finding container 9327e594c266f7dc98c242856f0a43c486d0f232b0b16b3e1fb1f19daa416297: Status 404 returned error can't find the container with id 9327e594c266f7dc98c242856f0a43c486d0f232b0b16b3e1fb1f19daa416297 Oct 31 01:00:01.321125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:01.321103 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb"] Oct 31 01:00:01.414739 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:00:01.414702 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14a9d7a_f8a3_43f5_a507_cdd72fa85df0.slice/crio-4fb4982d65719f419f9927e6481cc4c5903478c1731771724c78cb65d7aa5011 WatchSource:0}: Error finding container 4fb4982d65719f419f9927e6481cc4c5903478c1731771724c78cb65d7aa5011: Status 404 returned error can't find the container with id 4fb4982d65719f419f9927e6481cc4c5903478c1731771724c78cb65d7aa5011 Oct 31 01:00:01.416227 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:00:01.416199 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bcf361c_5cc7_483e_93f2_063c1e38e0df.slice/crio-666124935c2fd38dd05196a78808b2d7d10d5a190de63ae6c6aadb2c4e884e26 WatchSource:0}: Error finding container 666124935c2fd38dd05196a78808b2d7d10d5a190de63ae6c6aadb2c4e884e26: Status 404 returned error can't find the container with id 666124935c2fd38dd05196a78808b2d7d10d5a190de63ae6c6aadb2c4e884e26 Oct 31 01:00:02.259555 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.259513 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" event={"ID":"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0","Type":"ContainerStarted","Data":"4aa88d3c16b5024b3ac8b56b91d75d659a6618f4e408448002aef4eb5d96ffad"} Oct 31 01:00:02.260006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.259614 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" event={"ID":"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0","Type":"ContainerStarted","Data":"4fb4982d65719f419f9927e6481cc4c5903478c1731771724c78cb65d7aa5011"} Oct 31 01:00:02.261868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.261836 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" event={"ID":"b7a3049f-1270-4d3c-a8ad-f13ffe40a214","Type":"ContainerStarted","Data":"1f6070c7ab3267340c3fa98b6b101a7259af084c8f36ffd91a3845dafd1f34a6"} Oct 31 01:00:02.263614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.263587 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" event={"ID":"380d9685-5509-41ec-85ea-f4488a8e1461","Type":"ContainerStarted","Data":"ddcdea2b8d58374b3ce26d16b2d1fab4ffa8da50d7f1efbf495e46bc990668ae"} Oct 31 01:00:02.263728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.263636 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" event={"ID":"380d9685-5509-41ec-85ea-f4488a8e1461","Type":"ContainerStarted","Data":"9327e594c266f7dc98c242856f0a43c486d0f232b0b16b3e1fb1f19daa416297"} Oct 31 01:00:02.265377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.265350 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" event={"ID":"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9","Type":"ContainerStarted","Data":"3f6d1d2a4eb8cfaec126b6368fe5335eab1579e43ba4151c6b12a6557285f91a"} Oct 31 01:00:02.266411 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.265402 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" event={"ID":"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9","Type":"ContainerStarted","Data":"dfeadb4031c0d8fbf6e352902cfb583f2dc6d16b5b831ababf58ffd8164adcd2"} Oct 31 01:00:02.267077 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.267056 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" event={"ID":"ca573556-53d4-431a-8f75-1637ebaa618b","Type":"ContainerStarted","Data":"72d50d6645667e7143fe9078b77a375f4726ab97a092b7d8d70fc4efbc6067b8"} Oct 31 01:00:02.268603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.268582 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" event={"ID":"69ed1af6-7b4d-459d-86d6-799672dd7ded","Type":"ContainerStarted","Data":"ba1c41600e71b6c66d87f87c745d6df18f2764e210556c2777d3481a8ecdbc7f"} Oct 31 01:00:02.268692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.268620 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" event={"ID":"69ed1af6-7b4d-459d-86d6-799672dd7ded","Type":"ContainerStarted","Data":"a8c6a09056fc423143d896cea48fa6709ee40bf9b48eaa3403a7f7618eb1678d"} Oct 31 01:00:02.270171 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.270146 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" event={"ID":"c884e066-0089-4a02-a980-6bcf9b5c8e89","Type":"ContainerStarted","Data":"248c9c2e4d5d8361994b8c27207526ff1d275730a038fe11045c3c4370c45705"} Oct 31 01:00:02.271791 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.271769 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" event={"ID":"449f0be5-c787-4cb0-841d-96a71464d09a","Type":"ContainerStarted","Data":"ced7db427ecab746785f379ff31b5891cfe6793b5a11fb00672ba1999848085b"} Oct 31 01:00:02.271905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.271811 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" event={"ID":"449f0be5-c787-4cb0-841d-96a71464d09a","Type":"ContainerStarted","Data":"47359fd2117176a484b1b24de6ca163fba7b797998f609651e8319f45e448a35"} Oct 31 01:00:02.273290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.273264 5978 generic.go:356] "Generic (PLEG): container finished" podID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerID="adad3b28ab152f2ecb384a379e9c0c53fbcfd6360fedf4e4cb8938b8fda36da5" exitCode=60 Oct 31 01:00:02.273379 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.273346 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" event={"ID":"f1a039bc-2f7e-4626-afef-28baa7d75b44","Type":"ContainerDied","Data":"adad3b28ab152f2ecb384a379e9c0c53fbcfd6360fedf4e4cb8938b8fda36da5"} Oct 31 01:00:02.273570 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.273552 5978 scope.go:122] "RemoveContainer" containerID="adad3b28ab152f2ecb384a379e9c0c53fbcfd6360fedf4e4cb8938b8fda36da5" Oct 31 01:00:02.274866 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.274845 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" event={"ID":"d3d3f24c-70f4-4199-b581-d4190e64eb61","Type":"ContainerStarted","Data":"80579c034d2db70de6b03065c5e25826e021567f9c3bfb12fdca6e6fd580e4ab"} Oct 31 01:00:02.274958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.274880 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" event={"ID":"d3d3f24c-70f4-4199-b581-d4190e64eb61","Type":"ContainerStarted","Data":"ceac3ead4b995770320764b88b5fb2ba5e6a712c5978eaaf5b25b45211744272"} Oct 31 01:00:02.276580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.276559 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" event={"ID":"4bcf361c-5cc7-483e-93f2-063c1e38e0df","Type":"ContainerStarted","Data":"6d2fe09c9096ce7d3ce0df54f0e46f597ab6cf5aef0588ed703a23033340fff8"} Oct 31 01:00:02.276670 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.276591 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" event={"ID":"4bcf361c-5cc7-483e-93f2-063c1e38e0df","Type":"ContainerStarted","Data":"666124935c2fd38dd05196a78808b2d7d10d5a190de63ae6c6aadb2c4e884e26"} Oct 31 01:00:02.300065 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.299969 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" podStartSLOduration=2.299950567 podStartE2EDuration="2.299950567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.296428695 +0000 UTC m=+2419.994542459" watchObservedRunningTime="2026-10-31 01:00:02.299950567 +0000 UTC m=+2419.998064335" Oct 31 01:00:02.338496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.338427 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" podStartSLOduration=2.338413389 podStartE2EDuration="2.338413389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.333975649 +0000 UTC m=+2420.032089413" watchObservedRunningTime="2026-10-31 01:00:02.338413389 +0000 UTC m=+2420.036527170" Oct 31 01:00:02.372299 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.372152 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" podStartSLOduration=2.37213513 podStartE2EDuration="2.37213513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.369585133 +0000 UTC m=+2420.067698895" watchObservedRunningTime="2026-10-31 01:00:02.37213513 +0000 UTC m=+2420.070248895" Oct 31 01:00:02.445970 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.431915 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" podStartSLOduration=2.431893397 podStartE2EDuration="2.431893397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.430904092 +0000 UTC m=+2420.129017858" watchObservedRunningTime="2026-10-31 01:00:02.431893397 +0000 UTC m=+2420.130007163" Oct 31 01:00:02.456051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.455894 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:02.523326 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.523273 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" podStartSLOduration=2.523259027 podStartE2EDuration="2.523259027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.519501342 +0000 UTC m=+2420.217615101" watchObservedRunningTime="2026-10-31 01:00:02.523259027 +0000 UTC m=+2420.221372791" Oct 31 01:00:02.558320 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.558263 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" podStartSLOduration=2.558248174 podStartE2EDuration="2.558248174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.5552631 +0000 UTC m=+2420.253376864" watchObservedRunningTime="2026-10-31 01:00:02.558248174 +0000 UTC m=+2420.256361938" Oct 31 01:00:02.591760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.591694 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" podStartSLOduration=2.591673966 podStartE2EDuration="2.591673966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.589488408 +0000 UTC m=+2420.287602171" watchObservedRunningTime="2026-10-31 01:00:02.591673966 +0000 UTC m=+2420.289787741" Oct 31 01:00:02.669746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.669634 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" podStartSLOduration=2.669618595 podStartE2EDuration="2.669618595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.669023366 +0000 UTC m=+2420.367137131" watchObservedRunningTime="2026-10-31 01:00:02.669618595 +0000 UTC m=+2420.367732359" Oct 31 01:00:02.671929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.671883 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" podStartSLOduration=2.67187004 podStartE2EDuration="2.67187004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.631473548 +0000 UTC m=+2420.329587303" watchObservedRunningTime="2026-10-31 01:00:02.67187004 +0000 UTC m=+2420.369983805" Oct 31 01:00:02.707938 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:02.707882 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" podStartSLOduration=2.707867018 podStartE2EDuration="2.707867018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.707358195 +0000 UTC m=+2420.405472019" watchObservedRunningTime="2026-10-31 01:00:02.707867018 +0000 UTC m=+2420.405980783" Oct 31 01:00:03.184530 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:03.184416 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:00:03.287384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.287349 5978 generic.go:356] "Generic (PLEG): container finished" podID="ca573556-53d4-431a-8f75-1637ebaa618b" containerID="72d50d6645667e7143fe9078b77a375f4726ab97a092b7d8d70fc4efbc6067b8" exitCode=60 Oct 31 01:00:03.287860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.287437 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" event={"ID":"ca573556-53d4-431a-8f75-1637ebaa618b","Type":"ContainerDied","Data":"72d50d6645667e7143fe9078b77a375f4726ab97a092b7d8d70fc4efbc6067b8"} Oct 31 01:00:03.287860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.287780 5978 scope.go:122] "RemoveContainer" containerID="72d50d6645667e7143fe9078b77a375f4726ab97a092b7d8d70fc4efbc6067b8" Oct 31 01:00:03.289330 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.289302 5978 generic.go:356] "Generic (PLEG): container finished" podID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerID="ba1c41600e71b6c66d87f87c745d6df18f2764e210556c2777d3481a8ecdbc7f" exitCode=60 Oct 31 01:00:03.289489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.289412 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" event={"ID":"69ed1af6-7b4d-459d-86d6-799672dd7ded","Type":"ContainerDied","Data":"ba1c41600e71b6c66d87f87c745d6df18f2764e210556c2777d3481a8ecdbc7f"} Oct 31 01:00:03.289768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.289660 5978 scope.go:122] "RemoveContainer" containerID="ba1c41600e71b6c66d87f87c745d6df18f2764e210556c2777d3481a8ecdbc7f" Oct 31 01:00:03.291776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.291636 5978 generic.go:356] "Generic (PLEG): container finished" podID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerID="248c9c2e4d5d8361994b8c27207526ff1d275730a038fe11045c3c4370c45705" exitCode=60 Oct 31 01:00:03.291776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.291721 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" event={"ID":"c884e066-0089-4a02-a980-6bcf9b5c8e89","Type":"ContainerDied","Data":"248c9c2e4d5d8361994b8c27207526ff1d275730a038fe11045c3c4370c45705"} Oct 31 01:00:03.295369 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.292144 5978 scope.go:122] "RemoveContainer" containerID="248c9c2e4d5d8361994b8c27207526ff1d275730a038fe11045c3c4370c45705" Oct 31 01:00:03.295369 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.294839 5978 generic.go:356] "Generic (PLEG): container finished" podID="449f0be5-c787-4cb0-841d-96a71464d09a" containerID="ced7db427ecab746785f379ff31b5891cfe6793b5a11fb00672ba1999848085b" exitCode=60 Oct 31 01:00:03.295369 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.294950 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" event={"ID":"449f0be5-c787-4cb0-841d-96a71464d09a","Type":"ContainerDied","Data":"ced7db427ecab746785f379ff31b5891cfe6793b5a11fb00672ba1999848085b"} Oct 31 01:00:03.295517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.295374 5978 scope.go:122] "RemoveContainer" containerID="ced7db427ecab746785f379ff31b5891cfe6793b5a11fb00672ba1999848085b" Oct 31 01:00:03.298822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.298483 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" event={"ID":"f1a039bc-2f7e-4626-afef-28baa7d75b44","Type":"ContainerStarted","Data":"8334f00b47ed5f3cb46a05080341152694cf3db634420746d861e04f5d6b21ae"} Oct 31 01:00:03.300525 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.300494 5978 generic.go:356] "Generic (PLEG): container finished" podID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerID="80579c034d2db70de6b03065c5e25826e021567f9c3bfb12fdca6e6fd580e4ab" exitCode=60 Oct 31 01:00:03.300653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.300584 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" event={"ID":"d3d3f24c-70f4-4199-b581-d4190e64eb61","Type":"ContainerDied","Data":"80579c034d2db70de6b03065c5e25826e021567f9c3bfb12fdca6e6fd580e4ab"} Oct 31 01:00:03.300817 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.300797 5978 scope.go:122] "RemoveContainer" containerID="80579c034d2db70de6b03065c5e25826e021567f9c3bfb12fdca6e6fd580e4ab" Oct 31 01:00:03.302497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.302465 5978 generic.go:356] "Generic (PLEG): container finished" podID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerID="6d2fe09c9096ce7d3ce0df54f0e46f597ab6cf5aef0588ed703a23033340fff8" exitCode=60 Oct 31 01:00:03.302598 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.302553 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" event={"ID":"4bcf361c-5cc7-483e-93f2-063c1e38e0df","Type":"ContainerDied","Data":"6d2fe09c9096ce7d3ce0df54f0e46f597ab6cf5aef0588ed703a23033340fff8"} Oct 31 01:00:03.302779 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.302764 5978 scope.go:122] "RemoveContainer" containerID="6d2fe09c9096ce7d3ce0df54f0e46f597ab6cf5aef0588ed703a23033340fff8" Oct 31 01:00:03.306842 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.306729 5978 generic.go:356] "Generic (PLEG): container finished" podID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerID="4aa88d3c16b5024b3ac8b56b91d75d659a6618f4e408448002aef4eb5d96ffad" exitCode=60 Oct 31 01:00:03.306899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.306836 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" event={"ID":"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0","Type":"ContainerDied","Data":"4aa88d3c16b5024b3ac8b56b91d75d659a6618f4e408448002aef4eb5d96ffad"} Oct 31 01:00:03.307059 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.307036 5978 scope.go:122] "RemoveContainer" containerID="4aa88d3c16b5024b3ac8b56b91d75d659a6618f4e408448002aef4eb5d96ffad" Oct 31 01:00:03.309508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.308811 5978 generic.go:356] "Generic (PLEG): container finished" podID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerID="1f6070c7ab3267340c3fa98b6b101a7259af084c8f36ffd91a3845dafd1f34a6" exitCode=60 Oct 31 01:00:03.309508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.308888 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" event={"ID":"b7a3049f-1270-4d3c-a8ad-f13ffe40a214","Type":"ContainerDied","Data":"1f6070c7ab3267340c3fa98b6b101a7259af084c8f36ffd91a3845dafd1f34a6"} Oct 31 01:00:03.309508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.309114 5978 scope.go:122] "RemoveContainer" containerID="1f6070c7ab3267340c3fa98b6b101a7259af084c8f36ffd91a3845dafd1f34a6" Oct 31 01:00:03.311175 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.311156 5978 generic.go:356] "Generic (PLEG): container finished" podID="380d9685-5509-41ec-85ea-f4488a8e1461" containerID="ddcdea2b8d58374b3ce26d16b2d1fab4ffa8da50d7f1efbf495e46bc990668ae" exitCode=60 Oct 31 01:00:03.311465 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.311432 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" event={"ID":"380d9685-5509-41ec-85ea-f4488a8e1461","Type":"ContainerDied","Data":"ddcdea2b8d58374b3ce26d16b2d1fab4ffa8da50d7f1efbf495e46bc990668ae"} Oct 31 01:00:03.311838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.311822 5978 scope.go:122] "RemoveContainer" containerID="ddcdea2b8d58374b3ce26d16b2d1fab4ffa8da50d7f1efbf495e46bc990668ae" Oct 31 01:00:03.313618 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.313578 5978 generic.go:356] "Generic (PLEG): container finished" podID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerID="3f6d1d2a4eb8cfaec126b6368fe5335eab1579e43ba4151c6b12a6557285f91a" exitCode=60 Oct 31 01:00:03.313771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.313628 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" event={"ID":"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9","Type":"ContainerDied","Data":"3f6d1d2a4eb8cfaec126b6368fe5335eab1579e43ba4151c6b12a6557285f91a"} Oct 31 01:00:03.317474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.313943 5978 scope.go:122] "RemoveContainer" containerID="3f6d1d2a4eb8cfaec126b6368fe5335eab1579e43ba4151c6b12a6557285f91a" Oct 31 01:00:03.362577 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:03.362512 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" podStartSLOduration=3.3624922440000002 podStartE2EDuration="3.362492244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:03.358232177 +0000 UTC m=+2421.056345942" watchObservedRunningTime="2026-10-31 01:00:03.362492244 +0000 UTC m=+2421.060606012" Oct 31 01:00:04.253948 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.253901 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:04.337865 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.337835 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" event={"ID":"b7a3049f-1270-4d3c-a8ad-f13ffe40a214","Type":"ContainerStarted","Data":"6aa711ccc6b4bd3db9614de3a4cd30d954e11717363691efe0295c64b43b91ec"} Oct 31 01:00:04.340210 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.340180 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" event={"ID":"380d9685-5509-41ec-85ea-f4488a8e1461","Type":"ContainerStarted","Data":"7676d53657239fa17001b2ae72e330b364f2ee3a17b2a319dbd021a86daff748"} Oct 31 01:00:04.342360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.342337 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" event={"ID":"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9","Type":"ContainerStarted","Data":"e311cbb2c44f4af8a617b651dde20cd07e29ca677c4a1a0747f9c7959683142a"} Oct 31 01:00:04.344248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.344222 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" event={"ID":"ca573556-53d4-431a-8f75-1637ebaa618b","Type":"ContainerStarted","Data":"8ef271e6871d3e95659a15cefd4096ed01680665f088a1ba69ed0f25c065b78f"} Oct 31 01:00:04.346898 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.346875 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" event={"ID":"c884e066-0089-4a02-a980-6bcf9b5c8e89","Type":"ContainerStarted","Data":"bb1eea49e934f514a85f21ea1cb1e209c2d2d1b535c38380a61e028ec997c717"} Oct 31 01:00:04.350347 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.350322 5978 generic.go:356] "Generic (PLEG): container finished" podID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerID="8334f00b47ed5f3cb46a05080341152694cf3db634420746d861e04f5d6b21ae" exitCode=60 Oct 31 01:00:04.350423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.350401 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" event={"ID":"f1a039bc-2f7e-4626-afef-28baa7d75b44","Type":"ContainerDied","Data":"8334f00b47ed5f3cb46a05080341152694cf3db634420746d861e04f5d6b21ae"} Oct 31 01:00:04.350489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.350474 5978 scope.go:122] "RemoveContainer" containerID="adad3b28ab152f2ecb384a379e9c0c53fbcfd6360fedf4e4cb8938b8fda36da5" Oct 31 01:00:04.350623 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.350608 5978 scope.go:122] "RemoveContainer" containerID="8334f00b47ed5f3cb46a05080341152694cf3db634420746d861e04f5d6b21ae" Oct 31 01:00:04.350918 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:04.350896 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr_odh-ai-gateway-infra(f1a039bc-2f7e-4626-afef-28baa7d75b44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" Oct 31 01:00:04.352592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.352567 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" event={"ID":"d3d3f24c-70f4-4199-b581-d4190e64eb61","Type":"ContainerStarted","Data":"83176880d475407996914a21e96417f9cdfc93660010ad65f6b2471df497aceb"} Oct 31 01:00:04.354672 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.354649 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" event={"ID":"4bcf361c-5cc7-483e-93f2-063c1e38e0df","Type":"ContainerStarted","Data":"50abefa2708196f5f9570b6eae5bc207d40243a053f6175e950ee113f909326a"} Oct 31 01:00:04.356873 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:04.356845 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" event={"ID":"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0","Type":"ContainerStarted","Data":"50f5a1ef233c4406c7f5465fd9caa225cda787c976c36b36482d57f9246adbfd"} Oct 31 01:00:05.369169 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.369134 5978 generic.go:356] "Generic (PLEG): container finished" podID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerID="e311cbb2c44f4af8a617b651dde20cd07e29ca677c4a1a0747f9c7959683142a" exitCode=60 Oct 31 01:00:05.369613 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.369260 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" event={"ID":"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9","Type":"ContainerDied","Data":"e311cbb2c44f4af8a617b651dde20cd07e29ca677c4a1a0747f9c7959683142a"} Oct 31 01:00:05.369613 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.369314 5978 scope.go:122] "RemoveContainer" containerID="3f6d1d2a4eb8cfaec126b6368fe5335eab1579e43ba4151c6b12a6557285f91a" Oct 31 01:00:05.369768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.369711 5978 scope.go:122] "RemoveContainer" containerID="e311cbb2c44f4af8a617b651dde20cd07e29ca677c4a1a0747f9c7959683142a" Oct 31 01:00:05.370190 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.370165 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw_odh-ai-gateway-infra(7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" Oct 31 01:00:05.371641 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.371606 5978 generic.go:356] "Generic (PLEG): container finished" podID="ca573556-53d4-431a-8f75-1637ebaa618b" containerID="8ef271e6871d3e95659a15cefd4096ed01680665f088a1ba69ed0f25c065b78f" exitCode=60 Oct 31 01:00:05.371743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.371685 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" event={"ID":"ca573556-53d4-431a-8f75-1637ebaa618b","Type":"ContainerDied","Data":"8ef271e6871d3e95659a15cefd4096ed01680665f088a1ba69ed0f25c065b78f"} Oct 31 01:00:05.372008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.371992 5978 scope.go:122] "RemoveContainer" containerID="8ef271e6871d3e95659a15cefd4096ed01680665f088a1ba69ed0f25c065b78f" Oct 31 01:00:05.372404 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.372384 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc_odh-ai-gateway-infra(ca573556-53d4-431a-8f75-1637ebaa618b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" Oct 31 01:00:05.373603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.373579 5978 generic.go:356] "Generic (PLEG): container finished" podID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerID="ba22c3af267e4db41251f92103f20bf847948d6959a29838cbd8353a221e71df" exitCode=60 Oct 31 01:00:05.373696 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.373606 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" event={"ID":"69ed1af6-7b4d-459d-86d6-799672dd7ded","Type":"ContainerDied","Data":"ba22c3af267e4db41251f92103f20bf847948d6959a29838cbd8353a221e71df"} Oct 31 01:00:05.373965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.373947 5978 scope.go:122] "RemoveContainer" containerID="ba22c3af267e4db41251f92103f20bf847948d6959a29838cbd8353a221e71df" Oct 31 01:00:05.374331 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.374307 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr_odh-ai-gateway-infra(69ed1af6-7b4d-459d-86d6-799672dd7ded)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" Oct 31 01:00:05.375860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.375839 5978 generic.go:356] "Generic (PLEG): container finished" podID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerID="bb1eea49e934f514a85f21ea1cb1e209c2d2d1b535c38380a61e028ec997c717" exitCode=60 Oct 31 01:00:05.375933 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.375916 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" event={"ID":"c884e066-0089-4a02-a980-6bcf9b5c8e89","Type":"ContainerDied","Data":"bb1eea49e934f514a85f21ea1cb1e209c2d2d1b535c38380a61e028ec997c717"} Oct 31 01:00:05.376173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.376159 5978 scope.go:122] "RemoveContainer" containerID="bb1eea49e934f514a85f21ea1cb1e209c2d2d1b535c38380a61e028ec997c717" Oct 31 01:00:05.376547 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.376525 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h_odh-ai-gateway-infra(c884e066-0089-4a02-a980-6bcf9b5c8e89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" Oct 31 01:00:05.378012 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.377993 5978 generic.go:356] "Generic (PLEG): container finished" podID="449f0be5-c787-4cb0-841d-96a71464d09a" containerID="b4c6928f3e5018b81a742fb7befd5c997f88c0f40635c83ecc3977f16d988c53" exitCode=60 Oct 31 01:00:05.378093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.378055 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" event={"ID":"449f0be5-c787-4cb0-841d-96a71464d09a","Type":"ContainerDied","Data":"b4c6928f3e5018b81a742fb7befd5c997f88c0f40635c83ecc3977f16d988c53"} Oct 31 01:00:05.378386 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.378369 5978 scope.go:122] "RemoveContainer" containerID="b4c6928f3e5018b81a742fb7befd5c997f88c0f40635c83ecc3977f16d988c53" Oct 31 01:00:05.378701 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.378681 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb_odh-ai-gateway-infra(449f0be5-c787-4cb0-841d-96a71464d09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" Oct 31 01:00:05.380031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.380015 5978 scope.go:122] "RemoveContainer" containerID="8334f00b47ed5f3cb46a05080341152694cf3db634420746d861e04f5d6b21ae" Oct 31 01:00:05.380291 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.380262 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr_odh-ai-gateway-infra(f1a039bc-2f7e-4626-afef-28baa7d75b44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" Oct 31 01:00:05.381833 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.381809 5978 generic.go:356] "Generic (PLEG): container finished" podID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerID="83176880d475407996914a21e96417f9cdfc93660010ad65f6b2471df497aceb" exitCode=60 Oct 31 01:00:05.381924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.381892 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" event={"ID":"d3d3f24c-70f4-4199-b581-d4190e64eb61","Type":"ContainerDied","Data":"83176880d475407996914a21e96417f9cdfc93660010ad65f6b2471df497aceb"} Oct 31 01:00:05.382090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.382074 5978 scope.go:122] "RemoveContainer" containerID="83176880d475407996914a21e96417f9cdfc93660010ad65f6b2471df497aceb" Oct 31 01:00:05.382342 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.382320 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj_odh-ai-gateway-infra(d3d3f24c-70f4-4199-b581-d4190e64eb61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" Oct 31 01:00:05.383879 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.383862 5978 generic.go:356] "Generic (PLEG): container finished" podID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerID="50abefa2708196f5f9570b6eae5bc207d40243a053f6175e950ee113f909326a" exitCode=60 Oct 31 01:00:05.383957 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.383941 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" event={"ID":"4bcf361c-5cc7-483e-93f2-063c1e38e0df","Type":"ContainerDied","Data":"50abefa2708196f5f9570b6eae5bc207d40243a053f6175e950ee113f909326a"} Oct 31 01:00:05.384133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.384120 5978 scope.go:122] "RemoveContainer" containerID="50abefa2708196f5f9570b6eae5bc207d40243a053f6175e950ee113f909326a" Oct 31 01:00:05.384407 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.384385 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q_odh-ai-gateway-infra(4bcf361c-5cc7-483e-93f2-063c1e38e0df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" Oct 31 01:00:05.385645 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.385628 5978 scope.go:122] "RemoveContainer" containerID="72d50d6645667e7143fe9078b77a375f4726ab97a092b7d8d70fc4efbc6067b8" Oct 31 01:00:05.385946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.385923 5978 generic.go:356] "Generic (PLEG): container finished" podID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerID="50f5a1ef233c4406c7f5465fd9caa225cda787c976c36b36482d57f9246adbfd" exitCode=60 Oct 31 01:00:05.386079 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.386051 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" event={"ID":"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0","Type":"ContainerDied","Data":"50f5a1ef233c4406c7f5465fd9caa225cda787c976c36b36482d57f9246adbfd"} Oct 31 01:00:05.386215 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.386198 5978 scope.go:122] "RemoveContainer" containerID="50f5a1ef233c4406c7f5465fd9caa225cda787c976c36b36482d57f9246adbfd" Oct 31 01:00:05.386481 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.386444 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf_odh-ai-gateway-infra(c14a9d7a-f8a3-43f5-a507-cdd72fa85df0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" Oct 31 01:00:05.388098 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.388077 5978 generic.go:356] "Generic (PLEG): container finished" podID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerID="6aa711ccc6b4bd3db9614de3a4cd30d954e11717363691efe0295c64b43b91ec" exitCode=60 Oct 31 01:00:05.388190 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.388154 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" event={"ID":"b7a3049f-1270-4d3c-a8ad-f13ffe40a214","Type":"ContainerDied","Data":"6aa711ccc6b4bd3db9614de3a4cd30d954e11717363691efe0295c64b43b91ec"} Oct 31 01:00:05.388355 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.388342 5978 scope.go:122] "RemoveContainer" containerID="6aa711ccc6b4bd3db9614de3a4cd30d954e11717363691efe0295c64b43b91ec" Oct 31 01:00:05.388626 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.388605 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f_odh-ai-gateway-infra(b7a3049f-1270-4d3c-a8ad-f13ffe40a214)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" Oct 31 01:00:05.390274 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.390254 5978 generic.go:356] "Generic (PLEG): container finished" podID="380d9685-5509-41ec-85ea-f4488a8e1461" containerID="7676d53657239fa17001b2ae72e330b364f2ee3a17b2a319dbd021a86daff748" exitCode=60 Oct 31 01:00:05.390353 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.390285 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" event={"ID":"380d9685-5509-41ec-85ea-f4488a8e1461","Type":"ContainerDied","Data":"7676d53657239fa17001b2ae72e330b364f2ee3a17b2a319dbd021a86daff748"} Oct 31 01:00:05.390577 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.390551 5978 scope.go:122] "RemoveContainer" containerID="7676d53657239fa17001b2ae72e330b364f2ee3a17b2a319dbd021a86daff748" Oct 31 01:00:05.390807 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:05.390790 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx_odh-ai-gateway-infra(380d9685-5509-41ec-85ea-f4488a8e1461)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" Oct 31 01:00:05.401700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.401677 5978 scope.go:122] "RemoveContainer" containerID="ba1c41600e71b6c66d87f87c745d6df18f2764e210556c2777d3481a8ecdbc7f" Oct 31 01:00:05.417345 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.417316 5978 scope.go:122] "RemoveContainer" containerID="248c9c2e4d5d8361994b8c27207526ff1d275730a038fe11045c3c4370c45705" Oct 31 01:00:05.432930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.432910 5978 scope.go:122] "RemoveContainer" containerID="ced7db427ecab746785f379ff31b5891cfe6793b5a11fb00672ba1999848085b" Oct 31 01:00:05.450207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.450153 5978 scope.go:122] "RemoveContainer" containerID="80579c034d2db70de6b03065c5e25826e021567f9c3bfb12fdca6e6fd580e4ab" Oct 31 01:00:05.482461 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.482420 5978 scope.go:122] "RemoveContainer" containerID="6d2fe09c9096ce7d3ce0df54f0e46f597ab6cf5aef0588ed703a23033340fff8" Oct 31 01:00:05.499357 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.499333 5978 scope.go:122] "RemoveContainer" containerID="4aa88d3c16b5024b3ac8b56b91d75d659a6618f4e408448002aef4eb5d96ffad" Oct 31 01:00:05.514320 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.514297 5978 scope.go:122] "RemoveContainer" containerID="1f6070c7ab3267340c3fa98b6b101a7259af084c8f36ffd91a3845dafd1f34a6" Oct 31 01:00:05.533061 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:05.533038 5978 scope.go:122] "RemoveContainer" containerID="ddcdea2b8d58374b3ce26d16b2d1fab4ffa8da50d7f1efbf495e46bc990668ae" Oct 31 01:00:06.146401 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.145976 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:06.401032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.400940 5978 scope.go:122] "RemoveContainer" containerID="6aa711ccc6b4bd3db9614de3a4cd30d954e11717363691efe0295c64b43b91ec" Oct 31 01:00:06.401505 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:06.401425 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f_odh-ai-gateway-infra(b7a3049f-1270-4d3c-a8ad-f13ffe40a214)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" Oct 31 01:00:06.402786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.402763 5978 scope.go:122] "RemoveContainer" containerID="7676d53657239fa17001b2ae72e330b364f2ee3a17b2a319dbd021a86daff748" Oct 31 01:00:06.403154 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:06.403134 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx_odh-ai-gateway-infra(380d9685-5509-41ec-85ea-f4488a8e1461)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" Oct 31 01:00:06.404498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.404445 5978 scope.go:122] "RemoveContainer" containerID="e311cbb2c44f4af8a617b651dde20cd07e29ca677c4a1a0747f9c7959683142a" Oct 31 01:00:06.404859 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:06.404833 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw_odh-ai-gateway-infra(7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" Oct 31 01:00:06.405912 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.405895 5978 scope.go:122] "RemoveContainer" containerID="8ef271e6871d3e95659a15cefd4096ed01680665f088a1ba69ed0f25c065b78f" Oct 31 01:00:06.406204 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:06.406174 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc_odh-ai-gateway-infra(ca573556-53d4-431a-8f75-1637ebaa618b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" Oct 31 01:00:06.407572 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.407551 5978 scope.go:122] "RemoveContainer" containerID="ba22c3af267e4db41251f92103f20bf847948d6959a29838cbd8353a221e71df" Oct 31 01:00:06.407855 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:06.407832 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr_odh-ai-gateway-infra(69ed1af6-7b4d-459d-86d6-799672dd7ded)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" Oct 31 01:00:06.409067 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.409053 5978 scope.go:122] "RemoveContainer" containerID="bb1eea49e934f514a85f21ea1cb1e209c2d2d1b535c38380a61e028ec997c717" Oct 31 01:00:06.409335 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:06.409316 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h_odh-ai-gateway-infra(c884e066-0089-4a02-a980-6bcf9b5c8e89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" Oct 31 01:00:06.412140 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.412123 5978 scope.go:122] "RemoveContainer" containerID="83176880d475407996914a21e96417f9cdfc93660010ad65f6b2471df497aceb" Oct 31 01:00:06.412409 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:06.412387 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj_odh-ai-gateway-infra(d3d3f24c-70f4-4199-b581-d4190e64eb61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" Oct 31 01:00:06.413983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.413964 5978 scope.go:122] "RemoveContainer" containerID="50abefa2708196f5f9570b6eae5bc207d40243a053f6175e950ee113f909326a" Oct 31 01:00:06.414253 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:06.414233 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q_odh-ai-gateway-infra(4bcf361c-5cc7-483e-93f2-063c1e38e0df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" Oct 31 01:00:06.415382 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:06.415366 5978 scope.go:122] "RemoveContainer" containerID="50f5a1ef233c4406c7f5465fd9caa225cda787c976c36b36482d57f9246adbfd" Oct 31 01:00:06.415622 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:06.415602 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf_odh-ai-gateway-infra(c14a9d7a-f8a3-43f5-a507-cdd72fa85df0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" Oct 31 01:00:07.949614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:07.949580 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:09.885018 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:09.884979 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:11.619944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:11.619902 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:13.343490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:13.341082 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:15.029843 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:15.009277 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:15.417624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:15.417589 5978 scope.go:122] "RemoveContainer" containerID="7676d53657239fa17001b2ae72e330b364f2ee3a17b2a319dbd021a86daff748" Oct 31 01:00:15.417901 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:15.417881 5978 scope.go:122] "RemoveContainer" containerID="50f5a1ef233c4406c7f5465fd9caa225cda787c976c36b36482d57f9246adbfd" Oct 31 01:00:15.418048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:15.418020 5978 scope.go:122] "RemoveContainer" containerID="8334f00b47ed5f3cb46a05080341152694cf3db634420746d861e04f5d6b21ae" Oct 31 01:00:16.418784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:16.417956 5978 scope.go:122] "RemoveContainer" containerID="bb1eea49e934f514a85f21ea1cb1e209c2d2d1b535c38380a61e028ec997c717" Oct 31 01:00:16.419287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:16.419067 5978 scope.go:122] "RemoveContainer" containerID="50abefa2708196f5f9570b6eae5bc207d40243a053f6175e950ee113f909326a" Oct 31 01:00:16.509104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:16.509071 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" event={"ID":"380d9685-5509-41ec-85ea-f4488a8e1461","Type":"ContainerStarted","Data":"bc1631d23c5647703943226f89cebb6a9bf705abbe6b406c22924d9a49345150"} Oct 31 01:00:16.511355 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:16.511323 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" event={"ID":"f1a039bc-2f7e-4626-afef-28baa7d75b44","Type":"ContainerStarted","Data":"954f5fc918acc7f59b6bf411e59879d4cdb8da4f5048ceee676038fbba94a5e5"} Oct 31 01:00:16.514300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:16.514271 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" event={"ID":"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0","Type":"ContainerStarted","Data":"f6ced75f90ac3178f44d32a1331a515f4220f052464f62c21b42e79182cecb2b"} Oct 31 01:00:16.763119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:16.760965 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:17.417665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.417617 5978 scope.go:122] "RemoveContainer" containerID="e311cbb2c44f4af8a617b651dde20cd07e29ca677c4a1a0747f9c7959683142a" Oct 31 01:00:17.528776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.528688 5978 generic.go:356] "Generic (PLEG): container finished" podID="380d9685-5509-41ec-85ea-f4488a8e1461" containerID="bc1631d23c5647703943226f89cebb6a9bf705abbe6b406c22924d9a49345150" exitCode=60 Oct 31 01:00:17.528776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.528766 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" event={"ID":"380d9685-5509-41ec-85ea-f4488a8e1461","Type":"ContainerDied","Data":"bc1631d23c5647703943226f89cebb6a9bf705abbe6b406c22924d9a49345150"} Oct 31 01:00:17.529233 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.528860 5978 scope.go:122] "RemoveContainer" containerID="7676d53657239fa17001b2ae72e330b364f2ee3a17b2a319dbd021a86daff748" Oct 31 01:00:17.529290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.529278 5978 scope.go:122] "RemoveContainer" containerID="bc1631d23c5647703943226f89cebb6a9bf705abbe6b406c22924d9a49345150" Oct 31 01:00:17.529804 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:17.529773 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx_odh-ai-gateway-infra(380d9685-5509-41ec-85ea-f4488a8e1461)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" Oct 31 01:00:17.532125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.532098 5978 generic.go:356] "Generic (PLEG): container finished" podID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerID="6d78c4f0c81d005f70365d8bd02f7d6bb5d1903afb5e3f8768f28358b69dae7d" exitCode=60 Oct 31 01:00:17.532250 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.532135 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" event={"ID":"c884e066-0089-4a02-a980-6bcf9b5c8e89","Type":"ContainerDied","Data":"6d78c4f0c81d005f70365d8bd02f7d6bb5d1903afb5e3f8768f28358b69dae7d"} Oct 31 01:00:17.533004 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.532685 5978 scope.go:122] "RemoveContainer" containerID="6d78c4f0c81d005f70365d8bd02f7d6bb5d1903afb5e3f8768f28358b69dae7d" Oct 31 01:00:17.533280 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:17.533148 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h_odh-ai-gateway-infra(c884e066-0089-4a02-a980-6bcf9b5c8e89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" Oct 31 01:00:17.535150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.535122 5978 generic.go:356] "Generic (PLEG): container finished" podID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerID="954f5fc918acc7f59b6bf411e59879d4cdb8da4f5048ceee676038fbba94a5e5" exitCode=60 Oct 31 01:00:17.535267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.535233 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" event={"ID":"f1a039bc-2f7e-4626-afef-28baa7d75b44","Type":"ContainerDied","Data":"954f5fc918acc7f59b6bf411e59879d4cdb8da4f5048ceee676038fbba94a5e5"} Oct 31 01:00:17.536785 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.535937 5978 scope.go:122] "RemoveContainer" containerID="954f5fc918acc7f59b6bf411e59879d4cdb8da4f5048ceee676038fbba94a5e5" Oct 31 01:00:17.536785 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:17.536362 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr_odh-ai-gateway-infra(f1a039bc-2f7e-4626-afef-28baa7d75b44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" Oct 31 01:00:17.541119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.541093 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" event={"ID":"4bcf361c-5cc7-483e-93f2-063c1e38e0df","Type":"ContainerStarted","Data":"f9afee452086ea6b1354686ea7621127c63ab1821fa2a0e52869f59e0b37e386"} Oct 31 01:00:17.542186 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.541520 5978 scope.go:122] "RemoveContainer" containerID="f9afee452086ea6b1354686ea7621127c63ab1821fa2a0e52869f59e0b37e386" Oct 31 01:00:17.542186 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:17.541873 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q_odh-ai-gateway-infra(4bcf361c-5cc7-483e-93f2-063c1e38e0df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" Oct 31 01:00:17.543586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.543561 5978 generic.go:356] "Generic (PLEG): container finished" podID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerID="f6ced75f90ac3178f44d32a1331a515f4220f052464f62c21b42e79182cecb2b" exitCode=60 Oct 31 01:00:17.543688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.543608 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" event={"ID":"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0","Type":"ContainerDied","Data":"f6ced75f90ac3178f44d32a1331a515f4220f052464f62c21b42e79182cecb2b"} Oct 31 01:00:17.543982 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.543962 5978 scope.go:122] "RemoveContainer" containerID="f6ced75f90ac3178f44d32a1331a515f4220f052464f62c21b42e79182cecb2b" Oct 31 01:00:17.544280 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:17.544261 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf_odh-ai-gateway-infra(c14a9d7a-f8a3-43f5-a507-cdd72fa85df0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" Oct 31 01:00:17.561667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.561637 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx"] Oct 31 01:00:17.592280 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.592251 5978 scope.go:122] "RemoveContainer" containerID="bb1eea49e934f514a85f21ea1cb1e209c2d2d1b535c38380a61e028ec997c717" Oct 31 01:00:17.604972 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.604936 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr"] Oct 31 01:00:17.619202 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.619178 5978 scope.go:122] "RemoveContainer" containerID="8334f00b47ed5f3cb46a05080341152694cf3db634420746d861e04f5d6b21ae" Oct 31 01:00:17.641940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.641591 5978 scope.go:122] "RemoveContainer" containerID="50f5a1ef233c4406c7f5465fd9caa225cda787c976c36b36482d57f9246adbfd" Oct 31 01:00:17.654429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:17.654397 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf"] Oct 31 01:00:18.418583 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.418545 5978 scope.go:122] "RemoveContainer" containerID="8ef271e6871d3e95659a15cefd4096ed01680665f088a1ba69ed0f25c065b78f" Oct 31 01:00:18.581664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.581626 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:18.653375 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.651724 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" event={"ID":"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9","Type":"ContainerStarted","Data":"eb959c96e0db56bc14c59a6288fc514ff78de4cffd78f3a8ca8a1f5857a3a744"} Oct 31 01:00:18.653375 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.652187 5978 scope.go:122] "RemoveContainer" containerID="eb959c96e0db56bc14c59a6288fc514ff78de4cffd78f3a8ca8a1f5857a3a744" Oct 31 01:00:18.653375 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:18.652670 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw_odh-ai-gateway-infra(7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" Oct 31 01:00:18.666825 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.666796 5978 generic.go:356] "Generic (PLEG): container finished" podID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerID="f9afee452086ea6b1354686ea7621127c63ab1821fa2a0e52869f59e0b37e386" exitCode=60 Oct 31 01:00:18.666975 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.666919 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" event={"ID":"4bcf361c-5cc7-483e-93f2-063c1e38e0df","Type":"ContainerDied","Data":"f9afee452086ea6b1354686ea7621127c63ab1821fa2a0e52869f59e0b37e386"} Oct 31 01:00:18.667052 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.666976 5978 scope.go:122] "RemoveContainer" containerID="50abefa2708196f5f9570b6eae5bc207d40243a053f6175e950ee113f909326a" Oct 31 01:00:18.667669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.667521 5978 scope.go:122] "RemoveContainer" containerID="f9afee452086ea6b1354686ea7621127c63ab1821fa2a0e52869f59e0b37e386" Oct 31 01:00:18.669351 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:18.668968 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q_odh-ai-gateway-infra(4bcf361c-5cc7-483e-93f2-063c1e38e0df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" Oct 31 01:00:18.761365 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.750871 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q"] Oct 31 01:00:18.818290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.818254 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h"] Oct 31 01:00:18.957492 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:18.951561 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:19.081580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.081550 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:19.084506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.084395 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:19.091504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.090831 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380d9685-5509-41ec-85ea-f4488a8e1461-kube-api-access-g24gr\" (UniqueName: \"kubernetes.io/projected/380d9685-5509-41ec-85ea-f4488a8e1461-kube-api-access-g24gr\") pod \"380d9685-5509-41ec-85ea-f4488a8e1461\" (UID: \"380d9685-5509-41ec-85ea-f4488a8e1461\") " Oct 31 01:00:19.091504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.090905 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380d9685-5509-41ec-85ea-f4488a8e1461-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d9685-5509-41ec-85ea-f4488a8e1461-maas-api-service-ca\") pod \"380d9685-5509-41ec-85ea-f4488a8e1461\" (UID: \"380d9685-5509-41ec-85ea-f4488a8e1461\") " Oct 31 01:00:19.092950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.092800 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380d9685-5509-41ec-85ea-f4488a8e1461-maas-api-service-ca" pod "380d9685-5509-41ec-85ea-f4488a8e1461" (UID: "380d9685-5509-41ec-85ea-f4488a8e1461"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.094070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.093845 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380d9685-5509-41ec-85ea-f4488a8e1461-kube-api-access-g24gr" pod "380d9685-5509-41ec-85ea-f4488a8e1461" (UID: "380d9685-5509-41ec-85ea-f4488a8e1461"). InnerVolumeSpecName "kube-api-access-g24gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.122361 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.122271 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:19.194315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.194273 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a039bc-2f7e-4626-afef-28baa7d75b44-kube-api-access-xp8wv\" (UniqueName: \"kubernetes.io/projected/f1a039bc-2f7e-4626-afef-28baa7d75b44-kube-api-access-xp8wv\") pod \"f1a039bc-2f7e-4626-afef-28baa7d75b44\" (UID: \"f1a039bc-2f7e-4626-afef-28baa7d75b44\") " Oct 31 01:00:19.194548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.194367 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-kube-api-access-6cmlc\" (UniqueName: \"kubernetes.io/projected/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-kube-api-access-6cmlc\") pod \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\" (UID: \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\") " Oct 31 01:00:19.194723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.194703 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a039bc-2f7e-4626-afef-28baa7d75b44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a039bc-2f7e-4626-afef-28baa7d75b44-maas-api-service-ca\") pod \"f1a039bc-2f7e-4626-afef-28baa7d75b44\" (UID: \"f1a039bc-2f7e-4626-afef-28baa7d75b44\") " Oct 31 01:00:19.194801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.194761 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-maas-api-service-ca\") pod \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\" (UID: \"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0\") " Oct 31 01:00:19.195416 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.195396 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g24gr\" (UniqueName: \"kubernetes.io/projected/380d9685-5509-41ec-85ea-f4488a8e1461-kube-api-access-g24gr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.196339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.196303 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d9685-5509-41ec-85ea-f4488a8e1461-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.196572 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.196357 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-maas-api-service-ca" pod "c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" (UID: "c14a9d7a-f8a3-43f5-a507-cdd72fa85df0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.196687 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.196613 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a039bc-2f7e-4626-afef-28baa7d75b44-maas-api-service-ca" pod "f1a039bc-2f7e-4626-afef-28baa7d75b44" (UID: "f1a039bc-2f7e-4626-afef-28baa7d75b44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.200977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.200945 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-kube-api-access-6cmlc" pod "c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" (UID: "c14a9d7a-f8a3-43f5-a507-cdd72fa85df0"). InnerVolumeSpecName "kube-api-access-6cmlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.201538 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.201515 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a039bc-2f7e-4626-afef-28baa7d75b44-kube-api-access-xp8wv" pod "f1a039bc-2f7e-4626-afef-28baa7d75b44" (UID: "f1a039bc-2f7e-4626-afef-28baa7d75b44"). InnerVolumeSpecName "kube-api-access-xp8wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.297373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.297291 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c884e066-0089-4a02-a980-6bcf9b5c8e89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c884e066-0089-4a02-a980-6bcf9b5c8e89-maas-api-service-ca\") pod \"c884e066-0089-4a02-a980-6bcf9b5c8e89\" (UID: \"c884e066-0089-4a02-a980-6bcf9b5c8e89\") " Oct 31 01:00:19.297550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.297520 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c884e066-0089-4a02-a980-6bcf9b5c8e89-kube-api-access-q8wmc\" (UniqueName: \"kubernetes.io/projected/c884e066-0089-4a02-a980-6bcf9b5c8e89-kube-api-access-q8wmc\") pod \"c884e066-0089-4a02-a980-6bcf9b5c8e89\" (UID: \"c884e066-0089-4a02-a980-6bcf9b5c8e89\") " Oct 31 01:00:19.297752 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.297727 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c884e066-0089-4a02-a980-6bcf9b5c8e89-maas-api-service-ca" pod "c884e066-0089-4a02-a980-6bcf9b5c8e89" (UID: "c884e066-0089-4a02-a980-6bcf9b5c8e89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.298204 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.298180 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c884e066-0089-4a02-a980-6bcf9b5c8e89-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.298204 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.298204 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8wv\" (UniqueName: \"kubernetes.io/projected/f1a039bc-2f7e-4626-afef-28baa7d75b44-kube-api-access-xp8wv\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.298367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.298220 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cmlc\" (UniqueName: \"kubernetes.io/projected/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-kube-api-access-6cmlc\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.298367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.298234 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a039bc-2f7e-4626-afef-28baa7d75b44-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.298367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.298249 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.299682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.299662 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c884e066-0089-4a02-a980-6bcf9b5c8e89-kube-api-access-q8wmc" pod "c884e066-0089-4a02-a980-6bcf9b5c8e89" (UID: "c884e066-0089-4a02-a980-6bcf9b5c8e89"). InnerVolumeSpecName "kube-api-access-q8wmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.400259 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.400221 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8wmc\" (UniqueName: \"kubernetes.io/projected/c884e066-0089-4a02-a980-6bcf9b5c8e89-kube-api-access-q8wmc\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:19.418226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.418182 5978 scope.go:122] "RemoveContainer" containerID="6aa711ccc6b4bd3db9614de3a4cd30d954e11717363691efe0295c64b43b91ec" Oct 31 01:00:19.418555 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.418530 5978 scope.go:122] "RemoveContainer" containerID="b4c6928f3e5018b81a742fb7befd5c997f88c0f40635c83ecc3977f16d988c53" Oct 31 01:00:19.418708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.418660 5978 scope.go:122] "RemoveContainer" containerID="83176880d475407996914a21e96417f9cdfc93660010ad65f6b2471df497aceb" Oct 31 01:00:19.418782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.418754 5978 scope.go:122] "RemoveContainer" containerID="ba22c3af267e4db41251f92103f20bf847948d6959a29838cbd8353a221e71df" Oct 31 01:00:19.684592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.684564 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" Oct 31 01:00:19.685043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.684592 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx" event={"ID":"380d9685-5509-41ec-85ea-f4488a8e1461","Type":"ContainerDied","Data":"9327e594c266f7dc98c242856f0a43c486d0f232b0b16b3e1fb1f19daa416297"} Oct 31 01:00:19.685043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.684667 5978 scope.go:122] "RemoveContainer" containerID="bc1631d23c5647703943226f89cebb6a9bf705abbe6b406c22924d9a49345150" Oct 31 01:00:19.686959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.686927 5978 generic.go:356] "Generic (PLEG): container finished" podID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerID="eb959c96e0db56bc14c59a6288fc514ff78de4cffd78f3a8ca8a1f5857a3a744" exitCode=60 Oct 31 01:00:19.687079 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.686960 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" event={"ID":"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9","Type":"ContainerDied","Data":"eb959c96e0db56bc14c59a6288fc514ff78de4cffd78f3a8ca8a1f5857a3a744"} Oct 31 01:00:19.687495 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.687478 5978 scope.go:122] "RemoveContainer" containerID="eb959c96e0db56bc14c59a6288fc514ff78de4cffd78f3a8ca8a1f5857a3a744" Oct 31 01:00:19.687882 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:19.687862 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw_odh-ai-gateway-infra(7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" Oct 31 01:00:19.689505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.689424 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" event={"ID":"ca573556-53d4-431a-8f75-1637ebaa618b","Type":"ContainerStarted","Data":"4a64218bea83a6f3577e50acd549d50e5b7ba903678e7b455894977ebe2de4c5"} Oct 31 01:00:19.691176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.691149 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" Oct 31 01:00:19.691279 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.691195 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h" event={"ID":"c884e066-0089-4a02-a980-6bcf9b5c8e89","Type":"ContainerDied","Data":"1d2ef467748c741d2b2c046accddbe08cd63e8f10ea5b174a7a35514057227b0"} Oct 31 01:00:19.692965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.692934 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" Oct 31 01:00:19.692965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.692953 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr" event={"ID":"f1a039bc-2f7e-4626-afef-28baa7d75b44","Type":"ContainerDied","Data":"33d5472c6845a555b3a9e9a82d253d855f279744b28aba37736e3ece5303cc3e"} Oct 31 01:00:19.701738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.698341 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" Oct 31 01:00:19.701738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.698371 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf" event={"ID":"c14a9d7a-f8a3-43f5-a507-cdd72fa85df0","Type":"ContainerDied","Data":"4fb4982d65719f419f9927e6481cc4c5903478c1731771724c78cb65d7aa5011"} Oct 31 01:00:19.701738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.698832 5978 scope.go:122] "RemoveContainer" containerID="e311cbb2c44f4af8a617b651dde20cd07e29ca677c4a1a0747f9c7959683142a" Oct 31 01:00:19.735024 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.734991 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx"] Oct 31 01:00:19.743152 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.743114 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-82b5e8-29890140-vz4wx"] Oct 31 01:00:19.813082 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.813059 5978 scope.go:122] "RemoveContainer" containerID="6d78c4f0c81d005f70365d8bd02f7d6bb5d1903afb5e3f8768f28358b69dae7d" Oct 31 01:00:19.836287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.836261 5978 scope.go:122] "RemoveContainer" containerID="954f5fc918acc7f59b6bf411e59879d4cdb8da4f5048ceee676038fbba94a5e5" Oct 31 01:00:19.852080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.852053 5978 scope.go:122] "RemoveContainer" containerID="f6ced75f90ac3178f44d32a1331a515f4220f052464f62c21b42e79182cecb2b" Oct 31 01:00:19.865867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.865829 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw"] Oct 31 01:00:19.914384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.914356 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:19.934900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.934870 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf"] Oct 31 01:00:19.941653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.941622 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3c90d8-29890140-cfdmf"] Oct 31 01:00:19.979529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.979494 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h"] Oct 31 01:00:19.986991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:19.986954 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-609843-29890140-hzz2h"] Oct 31 01:00:20.010930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.010889 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bcf361c-5cc7-483e-93f2-063c1e38e0df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcf361c-5cc7-483e-93f2-063c1e38e0df-maas-api-service-ca\") pod \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\" (UID: \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\") " Oct 31 01:00:20.011154 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.011095 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcf361c-5cc7-483e-93f2-063c1e38e0df-kube-api-access-pzxvt\" (UniqueName: \"kubernetes.io/projected/4bcf361c-5cc7-483e-93f2-063c1e38e0df-kube-api-access-pzxvt\") pod \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\" (UID: \"4bcf361c-5cc7-483e-93f2-063c1e38e0df\") " Oct 31 01:00:20.011341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.011315 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcf361c-5cc7-483e-93f2-063c1e38e0df-maas-api-service-ca" pod "4bcf361c-5cc7-483e-93f2-063c1e38e0df" (UID: "4bcf361c-5cc7-483e-93f2-063c1e38e0df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:20.012172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.012147 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcf361c-5cc7-483e-93f2-063c1e38e0df-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:20.015813 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.015778 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcf361c-5cc7-483e-93f2-063c1e38e0df-kube-api-access-pzxvt" pod "4bcf361c-5cc7-483e-93f2-063c1e38e0df" (UID: "4bcf361c-5cc7-483e-93f2-063c1e38e0df"). InnerVolumeSpecName "kube-api-access-pzxvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:20.019829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.019801 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr"] Oct 31 01:00:20.024726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.024698 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-57e594-29890140-vcvnr"] Oct 31 01:00:20.114885 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.114840 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzxvt\" (UniqueName: \"kubernetes.io/projected/4bcf361c-5cc7-483e-93f2-063c1e38e0df-kube-api-access-pzxvt\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:20.430230 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.430191 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" path="/var/lib/kubelet/pods/380d9685-5509-41ec-85ea-f4488a8e1461/volumes" Oct 31 01:00:20.430776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.430760 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" path="/var/lib/kubelet/pods/c14a9d7a-f8a3-43f5-a507-cdd72fa85df0/volumes" Oct 31 01:00:20.431193 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.431180 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" path="/var/lib/kubelet/pods/c884e066-0089-4a02-a980-6bcf9b5c8e89/volumes" Oct 31 01:00:20.431936 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.431916 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" path="/var/lib/kubelet/pods/f1a039bc-2f7e-4626-afef-28baa7d75b44/volumes" Oct 31 01:00:20.533244 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.531096 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:20.589195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.589157 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847"] Oct 31 01:00:20.589406 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.589382 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" containerID="cri-o://131b4c353a547a5c37f72b3cf83aedf909dd243500aed904aeedadaa68e2edfe" gracePeriod=30 Oct 31 01:00:20.727673 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.727617 5978 generic.go:356] "Generic (PLEG): container finished" podID="ca573556-53d4-431a-8f75-1637ebaa618b" containerID="4a64218bea83a6f3577e50acd549d50e5b7ba903678e7b455894977ebe2de4c5" exitCode=60 Oct 31 01:00:20.728117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.727735 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" event={"ID":"ca573556-53d4-431a-8f75-1637ebaa618b","Type":"ContainerDied","Data":"4a64218bea83a6f3577e50acd549d50e5b7ba903678e7b455894977ebe2de4c5"} Oct 31 01:00:20.728117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.727794 5978 scope.go:122] "RemoveContainer" containerID="8ef271e6871d3e95659a15cefd4096ed01680665f088a1ba69ed0f25c065b78f" Oct 31 01:00:20.728289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.728268 5978 scope.go:122] "RemoveContainer" containerID="4a64218bea83a6f3577e50acd549d50e5b7ba903678e7b455894977ebe2de4c5" Oct 31 01:00:20.728833 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:20.728806 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc_odh-ai-gateway-infra(ca573556-53d4-431a-8f75-1637ebaa618b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" Oct 31 01:00:20.731553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.731522 5978 generic.go:356] "Generic (PLEG): container finished" podID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerID="cfa647bb22fdb8fe1f105da4a0841324dcc0c960d89e7cb1178c7f15b26b60b0" exitCode=60 Oct 31 01:00:20.731670 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.731590 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" event={"ID":"69ed1af6-7b4d-459d-86d6-799672dd7ded","Type":"ContainerDied","Data":"cfa647bb22fdb8fe1f105da4a0841324dcc0c960d89e7cb1178c7f15b26b60b0"} Oct 31 01:00:20.732168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.732153 5978 scope.go:122] "RemoveContainer" containerID="cfa647bb22fdb8fe1f105da4a0841324dcc0c960d89e7cb1178c7f15b26b60b0" Oct 31 01:00:20.732562 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:20.732540 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr_odh-ai-gateway-infra(69ed1af6-7b4d-459d-86d6-799672dd7ded)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" Oct 31 01:00:20.738739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.738710 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" event={"ID":"449f0be5-c787-4cb0-841d-96a71464d09a","Type":"ContainerStarted","Data":"1b61335e3f3ca6c41c65817f770b42810525dcfaeae6ad9a22bad3de86364166"} Oct 31 01:00:20.739326 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.739304 5978 scope.go:122] "RemoveContainer" containerID="1b61335e3f3ca6c41c65817f770b42810525dcfaeae6ad9a22bad3de86364166" Oct 31 01:00:20.739847 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:20.739818 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb_odh-ai-gateway-infra(449f0be5-c787-4cb0-841d-96a71464d09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" Oct 31 01:00:20.742591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.742569 5978 generic.go:356] "Generic (PLEG): container finished" podID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerID="9f56450c8352c87f8c1d80215bd05df5c600a92443dbf6f2047b803680191fa1" exitCode=60 Oct 31 01:00:20.742677 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.742593 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" event={"ID":"d3d3f24c-70f4-4199-b581-d4190e64eb61","Type":"ContainerDied","Data":"9f56450c8352c87f8c1d80215bd05df5c600a92443dbf6f2047b803680191fa1"} Oct 31 01:00:20.743103 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.743080 5978 scope.go:122] "RemoveContainer" containerID="9f56450c8352c87f8c1d80215bd05df5c600a92443dbf6f2047b803680191fa1" Oct 31 01:00:20.743532 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:20.743498 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj_odh-ai-gateway-infra(d3d3f24c-70f4-4199-b581-d4190e64eb61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" Oct 31 01:00:20.744531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.744509 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" Oct 31 01:00:20.744627 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.744598 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q" event={"ID":"4bcf361c-5cc7-483e-93f2-063c1e38e0df","Type":"ContainerDied","Data":"666124935c2fd38dd05196a78808b2d7d10d5a190de63ae6c6aadb2c4e884e26"} Oct 31 01:00:20.749575 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.749546 5978 generic.go:356] "Generic (PLEG): container finished" podID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerID="bf74290013f8ad5394a2af8feaf9a2fab25117d21587a6f936bce63e02c2e03a" exitCode=60 Oct 31 01:00:20.749707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.749607 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" event={"ID":"b7a3049f-1270-4d3c-a8ad-f13ffe40a214","Type":"ContainerDied","Data":"bf74290013f8ad5394a2af8feaf9a2fab25117d21587a6f936bce63e02c2e03a"} Oct 31 01:00:20.750107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.750089 5978 scope.go:122] "RemoveContainer" containerID="bf74290013f8ad5394a2af8feaf9a2fab25117d21587a6f936bce63e02c2e03a" Oct 31 01:00:20.750591 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:20.750543 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f_odh-ai-gateway-infra(b7a3049f-1270-4d3c-a8ad-f13ffe40a214)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" Oct 31 01:00:20.784205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.784177 5978 scope.go:122] "RemoveContainer" containerID="ba22c3af267e4db41251f92103f20bf847948d6959a29838cbd8353a221e71df" Oct 31 01:00:20.804787 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.804762 5978 scope.go:122] "RemoveContainer" containerID="83176880d475407996914a21e96417f9cdfc93660010ad65f6b2471df497aceb" Oct 31 01:00:20.854692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.854657 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc"] Oct 31 01:00:20.901326 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.901297 5978 scope.go:122] "RemoveContainer" containerID="f9afee452086ea6b1354686ea7621127c63ab1821fa2a0e52869f59e0b37e386" Oct 31 01:00:20.953516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.953443 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:20.954387 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:20.954368 5978 scope.go:122] "RemoveContainer" containerID="6aa711ccc6b4bd3db9614de3a4cd30d954e11717363691efe0295c64b43b91ec" Oct 31 01:00:21.016239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.016201 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l"] Oct 31 01:00:21.016435 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.016414 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" containerID="cri-o://32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca" gracePeriod=30 Oct 31 01:00:21.042028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.041996 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-kube-api-access-tlvvn\" (UniqueName: \"kubernetes.io/projected/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-kube-api-access-tlvvn\") pod \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\" (UID: \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\") " Oct 31 01:00:21.042230 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.042116 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-maas-api-service-ca\") pod \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\" (UID: \"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9\") " Oct 31 01:00:21.042822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.042789 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-maas-api-service-ca" pod "7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" (UID: "7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:21.042988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.042949 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:21.047213 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.047167 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-kube-api-access-tlvvn" pod "7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" (UID: "7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9"). InnerVolumeSpecName "kube-api-access-tlvvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:21.057930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.057866 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q"] Oct 31 01:00:21.062621 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.062593 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-498580-29890140-4j55q"] Oct 31 01:00:21.145896 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.145837 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlvvn\" (UniqueName: \"kubernetes.io/projected/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9-kube-api-access-tlvvn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:21.777762 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.777708 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" event={"ID":"7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9","Type":"ContainerDied","Data":"dfeadb4031c0d8fbf6e352902cfb583f2dc6d16b5b831ababf58ffd8164adcd2"} Oct 31 01:00:21.778272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.777792 5978 scope.go:122] "RemoveContainer" containerID="eb959c96e0db56bc14c59a6288fc514ff78de4cffd78f3a8ca8a1f5857a3a744" Oct 31 01:00:21.778272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.777900 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw" Oct 31 01:00:21.788521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.788476 5978 generic.go:356] "Generic (PLEG): container finished" podID="449f0be5-c787-4cb0-841d-96a71464d09a" containerID="1b61335e3f3ca6c41c65817f770b42810525dcfaeae6ad9a22bad3de86364166" exitCode=60 Oct 31 01:00:21.788692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.788617 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" event={"ID":"449f0be5-c787-4cb0-841d-96a71464d09a","Type":"ContainerDied","Data":"1b61335e3f3ca6c41c65817f770b42810525dcfaeae6ad9a22bad3de86364166"} Oct 31 01:00:21.789174 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.789143 5978 scope.go:122] "RemoveContainer" containerID="1b61335e3f3ca6c41c65817f770b42810525dcfaeae6ad9a22bad3de86364166" Oct 31 01:00:21.789746 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:21.789713 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb_odh-ai-gateway-infra(449f0be5-c787-4cb0-841d-96a71464d09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" Oct 31 01:00:21.795474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.795434 5978 scope.go:122] "RemoveContainer" containerID="b4c6928f3e5018b81a742fb7befd5c997f88c0f40635c83ecc3977f16d988c53" Oct 31 01:00:21.872681 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.872638 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr"] Oct 31 01:00:21.918534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.918492 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb"] Oct 31 01:00:21.949264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:21.949235 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj"] Oct 31 01:00:22.028805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.028589 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw"] Oct 31 01:00:22.028986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.028726 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:22.043253 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.043214 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890140-l7rrw"] Oct 31 01:00:22.050709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.050643 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f"] Oct 31 01:00:22.071766 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.071718 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca573556-53d4-431a-8f75-1637ebaa618b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca573556-53d4-431a-8f75-1637ebaa618b-maas-api-service-ca\") pod \"ca573556-53d4-431a-8f75-1637ebaa618b\" (UID: \"ca573556-53d4-431a-8f75-1637ebaa618b\") " Oct 31 01:00:22.071933 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.071793 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca573556-53d4-431a-8f75-1637ebaa618b-kube-api-access-j6ddp\" (UniqueName: \"kubernetes.io/projected/ca573556-53d4-431a-8f75-1637ebaa618b-kube-api-access-j6ddp\") pod \"ca573556-53d4-431a-8f75-1637ebaa618b\" (UID: \"ca573556-53d4-431a-8f75-1637ebaa618b\") " Oct 31 01:00:22.072490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.072240 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca573556-53d4-431a-8f75-1637ebaa618b-maas-api-service-ca" pod "ca573556-53d4-431a-8f75-1637ebaa618b" (UID: "ca573556-53d4-431a-8f75-1637ebaa618b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:22.072722 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.072704 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca573556-53d4-431a-8f75-1637ebaa618b-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.075729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.075691 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca573556-53d4-431a-8f75-1637ebaa618b-kube-api-access-j6ddp" pod "ca573556-53d4-431a-8f75-1637ebaa618b" (UID: "ca573556-53d4-431a-8f75-1637ebaa618b"). InnerVolumeSpecName "kube-api-access-j6ddp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:22.105591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.105554 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz"] Oct 31 01:00:22.106053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.105816 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" containerID="cri-o://ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4" gracePeriod=30 Oct 31 01:00:22.139338 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.139307 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:22.178603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.178556 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm"] Oct 31 01:00:22.196500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.179057 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ed1af6-7b4d-459d-86d6-799672dd7ded-kube-api-access-7mg8c\" (UniqueName: \"kubernetes.io/projected/69ed1af6-7b4d-459d-86d6-799672dd7ded-kube-api-access-7mg8c\") pod \"69ed1af6-7b4d-459d-86d6-799672dd7ded\" (UID: \"69ed1af6-7b4d-459d-86d6-799672dd7ded\") " Oct 31 01:00:22.196500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.196191 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" containerID="cri-o://2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162" gracePeriod=30 Oct 31 01:00:22.196500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.196263 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ed1af6-7b4d-459d-86d6-799672dd7ded-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ed1af6-7b4d-459d-86d6-799672dd7ded-maas-api-service-ca\") pod \"69ed1af6-7b4d-459d-86d6-799672dd7ded\" (UID: \"69ed1af6-7b4d-459d-86d6-799672dd7ded\") " Oct 31 01:00:22.197920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.197595 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ed1af6-7b4d-459d-86d6-799672dd7ded-maas-api-service-ca" pod "69ed1af6-7b4d-459d-86d6-799672dd7ded" (UID: "69ed1af6-7b4d-459d-86d6-799672dd7ded"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:22.197920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.197866 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ed1af6-7b4d-459d-86d6-799672dd7ded-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.197920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.197892 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6ddp\" (UniqueName: \"kubernetes.io/projected/ca573556-53d4-431a-8f75-1637ebaa618b-kube-api-access-j6ddp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.219627 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.219578 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ed1af6-7b4d-459d-86d6-799672dd7ded-kube-api-access-7mg8c" pod "69ed1af6-7b4d-459d-86d6-799672dd7ded" (UID: "69ed1af6-7b4d-459d-86d6-799672dd7ded"). InnerVolumeSpecName "kube-api-access-7mg8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:22.244359 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.244326 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:22.312305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.311774 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d3f24c-70f4-4199-b581-d4190e64eb61-kube-api-access-rxqjz\" (UniqueName: \"kubernetes.io/projected/d3d3f24c-70f4-4199-b581-d4190e64eb61-kube-api-access-rxqjz\") pod \"d3d3f24c-70f4-4199-b581-d4190e64eb61\" (UID: \"d3d3f24c-70f4-4199-b581-d4190e64eb61\") " Oct 31 01:00:22.312305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.312001 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d3f24c-70f4-4199-b581-d4190e64eb61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d3f24c-70f4-4199-b581-d4190e64eb61-maas-api-service-ca\") pod \"d3d3f24c-70f4-4199-b581-d4190e64eb61\" (UID: \"d3d3f24c-70f4-4199-b581-d4190e64eb61\") " Oct 31 01:00:22.313363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.313288 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d3f24c-70f4-4199-b581-d4190e64eb61-maas-api-service-ca" pod "d3d3f24c-70f4-4199-b581-d4190e64eb61" (UID: "d3d3f24c-70f4-4199-b581-d4190e64eb61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:22.316094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.314547 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mg8c\" (UniqueName: \"kubernetes.io/projected/69ed1af6-7b4d-459d-86d6-799672dd7ded-kube-api-access-7mg8c\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.316094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.314581 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d3f24c-70f4-4199-b581-d4190e64eb61-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.322660 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.322624 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d3f24c-70f4-4199-b581-d4190e64eb61-kube-api-access-rxqjz" pod "d3d3f24c-70f4-4199-b581-d4190e64eb61" (UID: "d3d3f24c-70f4-4199-b581-d4190e64eb61"). InnerVolumeSpecName "kube-api-access-rxqjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:22.332200 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.332168 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:22.415767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.415734 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-maas-api-service-ca\") pod \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\" (UID: \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\") " Oct 31 01:00:22.415964 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.415824 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-kube-api-access-7pg4c\" (UniqueName: \"kubernetes.io/projected/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-kube-api-access-7pg4c\") pod \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\" (UID: \"b7a3049f-1270-4d3c-a8ad-f13ffe40a214\") " Oct 31 01:00:22.416238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.416198 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-maas-api-service-ca" pod "b7a3049f-1270-4d3c-a8ad-f13ffe40a214" (UID: "b7a3049f-1270-4d3c-a8ad-f13ffe40a214"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:22.416652 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.416626 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.416652 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.416647 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxqjz\" (UniqueName: \"kubernetes.io/projected/d3d3f24c-70f4-4199-b581-d4190e64eb61-kube-api-access-rxqjz\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.418142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.418112 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-kube-api-access-7pg4c" pod "b7a3049f-1270-4d3c-a8ad-f13ffe40a214" (UID: "b7a3049f-1270-4d3c-a8ad-f13ffe40a214"). InnerVolumeSpecName "kube-api-access-7pg4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:22.428017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.427983 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" path="/var/lib/kubelet/pods/4bcf361c-5cc7-483e-93f2-063c1e38e0df/volumes" Oct 31 01:00:22.428521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.428505 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" path="/var/lib/kubelet/pods/7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9/volumes" Oct 31 01:00:22.518706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.518665 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pg4c\" (UniqueName: \"kubernetes.io/projected/b7a3049f-1270-4d3c-a8ad-f13ffe40a214-kube-api-access-7pg4c\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:22.561860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.561800 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:22.802848 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.802816 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" Oct 31 01:00:22.803270 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.802847 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f" event={"ID":"b7a3049f-1270-4d3c-a8ad-f13ffe40a214","Type":"ContainerDied","Data":"9b66f76e46b774610f1c6e043bb3cae6e6d2a58a3d46157d8ddf5e763a446331"} Oct 31 01:00:22.803270 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.802911 5978 scope.go:122] "RemoveContainer" containerID="bf74290013f8ad5394a2af8feaf9a2fab25117d21587a6f936bce63e02c2e03a" Oct 31 01:00:22.806053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.806020 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" event={"ID":"ca573556-53d4-431a-8f75-1637ebaa618b","Type":"ContainerDied","Data":"488a6da16a21c5ebb102f6697569b58adcd51ddda9db4281dbc9116e18d913e6"} Oct 31 01:00:22.806227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.806033 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc" Oct 31 01:00:22.807668 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.807645 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" Oct 31 01:00:22.807787 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.807660 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr" event={"ID":"69ed1af6-7b4d-459d-86d6-799672dd7ded","Type":"ContainerDied","Data":"a8c6a09056fc423143d896cea48fa6709ee40bf9b48eaa3403a7f7618eb1678d"} Oct 31 01:00:22.811209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.811189 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" Oct 31 01:00:22.811380 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.811352 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj" event={"ID":"d3d3f24c-70f4-4199-b581-d4190e64eb61","Type":"ContainerDied","Data":"ceac3ead4b995770320764b88b5fb2ba5e6a712c5978eaaf5b25b45211744272"} Oct 31 01:00:22.816293 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.816276 5978 scope.go:122] "RemoveContainer" containerID="4a64218bea83a6f3577e50acd549d50e5b7ba903678e7b455894977ebe2de4c5" Oct 31 01:00:22.829180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.829147 5978 scope.go:122] "RemoveContainer" containerID="cfa647bb22fdb8fe1f105da4a0841324dcc0c960d89e7cb1178c7f15b26b60b0" Oct 31 01:00:22.841332 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.841311 5978 scope.go:122] "RemoveContainer" containerID="9f56450c8352c87f8c1d80215bd05df5c600a92443dbf6f2047b803680191fa1" Oct 31 01:00:22.859343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.859311 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr"] Oct 31 01:00:22.866732 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.866697 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72120c-29890140-748dr"] Oct 31 01:00:22.936592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.936550 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj"] Oct 31 01:00:22.947042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.947007 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c942c9-29890140-vqkrj"] Oct 31 01:00:22.983154 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.983117 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc"] Oct 31 01:00:22.988946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:22.988921 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b87c62-29890140-5h2lc"] Oct 31 01:00:23.003037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.003002 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:23.021603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.021568 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f"] Oct 31 01:00:23.027624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.027592 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-762753-29890140-k998f"] Oct 31 01:00:23.135284 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.135232 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/449f0be5-c787-4cb0-841d-96a71464d09a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f0be5-c787-4cb0-841d-96a71464d09a-maas-api-service-ca\") pod \"449f0be5-c787-4cb0-841d-96a71464d09a\" (UID: \"449f0be5-c787-4cb0-841d-96a71464d09a\") " Oct 31 01:00:23.135489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.135335 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449f0be5-c787-4cb0-841d-96a71464d09a-kube-api-access-sjwbp\" (UniqueName: \"kubernetes.io/projected/449f0be5-c787-4cb0-841d-96a71464d09a-kube-api-access-sjwbp\") pod \"449f0be5-c787-4cb0-841d-96a71464d09a\" (UID: \"449f0be5-c787-4cb0-841d-96a71464d09a\") " Oct 31 01:00:23.135705 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.135678 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449f0be5-c787-4cb0-841d-96a71464d09a-maas-api-service-ca" pod "449f0be5-c787-4cb0-841d-96a71464d09a" (UID: "449f0be5-c787-4cb0-841d-96a71464d09a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:23.136219 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.136195 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f0be5-c787-4cb0-841d-96a71464d09a-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:23.137921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.137892 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449f0be5-c787-4cb0-841d-96a71464d09a-kube-api-access-sjwbp" pod "449f0be5-c787-4cb0-841d-96a71464d09a" (UID: "449f0be5-c787-4cb0-841d-96a71464d09a"). InnerVolumeSpecName "kube-api-access-sjwbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:23.238322 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.238285 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjwbp\" (UniqueName: \"kubernetes.io/projected/449f0be5-c787-4cb0-841d-96a71464d09a-kube-api-access-sjwbp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:23.313116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.312963 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:23.546995 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.546906 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk"] Oct 31 01:00:23.547946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.547921 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerName="cleanup" Oct 31 01:00:23.547946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.547943 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.547960 5978 cpu_manager.go:414] "containerMap: removing container" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.547965 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.547994 5978 cpu_manager.go:414] "containerMap: removing container" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548005 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548020 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548025 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548034 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548039 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548048 5978 cpu_manager.go:414] "containerMap: removing container" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548053 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548061 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548065 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548071 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548075 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548080 5978 cpu_manager.go:414] "containerMap: removing container" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548084 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548093 5978 cpu_manager.go:414] "containerMap: removing container" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548097 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548104 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548109 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548115 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548119 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548132 5978 cpu_manager.go:414] "containerMap: removing container" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548136 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerName="cleanup" Oct 31 01:00:23.548134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548144 5978 cpu_manager.go:414] "containerMap: removing container" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548149 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548155 5978 cpu_manager.go:414] "containerMap: removing container" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548159 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548167 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548171 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548182 5978 cpu_manager.go:414] "containerMap: removing container" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548186 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548196 5978 cpu_manager.go:414] "containerMap: removing container" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548199 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548212 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548217 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548225 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548229 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548239 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548243 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548254 5978 cpu_manager.go:414] "containerMap: removing container" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548258 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548267 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548271 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548277 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548281 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548287 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548291 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548297 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548301 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548314 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548318 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548330 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548334 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548345 5978 cpu_manager.go:414] "containerMap: removing container" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548360 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548366 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548371 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548521 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548532 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548538 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548545 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548551 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548557 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548565 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548571 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548577 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548583 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548588 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548593 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548600 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548607 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a039bc-2f7e-4626-afef-28baa7d75b44" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548615 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548619 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548625 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548633 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548639 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548645 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548651 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548657 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548662 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548668 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548674 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548679 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548684 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548690 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548696 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerName="cleanup" Oct 31 01:00:23.548882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.548701 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4b1f4c-2e52-43de-b9cd-b7496cbfa3c9" containerName="cleanup" Oct 31 01:00:23.553103 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.553083 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.566036 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.566008 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:00:23.572028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.571962 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk"] Oct 31 01:00:23.652581 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.652537 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swk24\" (UniqueName: \"kubernetes.io/projected/ed858859-aac8-4b37-9581-eb23c59b9ec6-kube-api-access-swk24\") pod \"maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.652781 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.652610 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.652781 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.652721 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.754384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.754353 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swk24\" (UniqueName: \"kubernetes.io/projected/ed858859-aac8-4b37-9581-eb23c59b9ec6-kube-api-access-swk24\") pod \"maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.754592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.754406 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.754592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.754444 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.755153 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.755130 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.756828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.756805 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.776079 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.776011 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swk24\" (UniqueName: \"kubernetes.io/projected/ed858859-aac8-4b37-9581-eb23c59b9ec6-kube-api-access-swk24\") pod \"maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.822519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.822446 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" Oct 31 01:00:23.823029 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.822480 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb" event={"ID":"449f0be5-c787-4cb0-841d-96a71464d09a","Type":"ContainerDied","Data":"47359fd2117176a484b1b24de6ca163fba7b797998f609651e8319f45e448a35"} Oct 31 01:00:23.823029 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.822642 5978 scope.go:122] "RemoveContainer" containerID="1b61335e3f3ca6c41c65817f770b42810525dcfaeae6ad9a22bad3de86364166" Oct 31 01:00:23.868354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.868317 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:23.875224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.875189 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb"] Oct 31 01:00:23.883472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:23.883204 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f8856-29890140-c6xbb"] Oct 31 01:00:24.183889 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.183855 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk"] Oct 31 01:00:24.184407 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:00:24.184378 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded858859_aac8_4b37_9581_eb23c59b9ec6.slice/crio-924489e645fe66952823dfe032511c75955ca2bca8666bcc03542bcc3d837c03 WatchSource:0}: Error finding container 924489e645fe66952823dfe032511c75955ca2bca8666bcc03542bcc3d837c03: Status 404 returned error can't find the container with id 924489e645fe66952823dfe032511c75955ca2bca8666bcc03542bcc3d837c03 Oct 31 01:00:24.428777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.428744 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="449f0be5-c787-4cb0-841d-96a71464d09a" path="/var/lib/kubelet/pods/449f0be5-c787-4cb0-841d-96a71464d09a/volumes" Oct 31 01:00:24.429273 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.429258 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ed1af6-7b4d-459d-86d6-799672dd7ded" path="/var/lib/kubelet/pods/69ed1af6-7b4d-459d-86d6-799672dd7ded/volumes" Oct 31 01:00:24.429767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.429750 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" path="/var/lib/kubelet/pods/b7a3049f-1270-4d3c-a8ad-f13ffe40a214/volumes" Oct 31 01:00:24.430551 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.430512 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" path="/var/lib/kubelet/pods/ca573556-53d4-431a-8f75-1637ebaa618b/volumes" Oct 31 01:00:24.430942 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.430929 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d3f24c-70f4-4199-b581-d4190e64eb61" path="/var/lib/kubelet/pods/d3d3f24c-70f4-4199-b581-d4190e64eb61/volumes" Oct 31 01:00:24.557479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.555999 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:24.839385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.839347 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" event={"ID":"ed858859-aac8-4b37-9581-eb23c59b9ec6","Type":"ContainerStarted","Data":"41c61425dff61151f6aaa526b4c2e26f623ef56d26db2a80b8ad47e6a50b7ffc"} Oct 31 01:00:24.839935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.839416 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" event={"ID":"ed858859-aac8-4b37-9581-eb23c59b9ec6","Type":"ContainerStarted","Data":"924489e645fe66952823dfe032511c75955ca2bca8666bcc03542bcc3d837c03"} Oct 31 01:00:24.879045 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:24.878874 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" podStartSLOduration=1.8788551020000002 podStartE2EDuration="1.878855102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:24.874880693 +0000 UTC m=+2442.572994460" watchObservedRunningTime="2026-10-31 01:00:24.878855102 +0000 UTC m=+2442.576968867" Oct 31 01:00:25.665286 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.665251 5978 patch_prober.go:29] interesting pod/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:00:25.665483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.665318 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 01:00:25.855173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.855095 5978 generic.go:356] "Generic (PLEG): container finished" podID="ed858859-aac8-4b37-9581-eb23c59b9ec6" containerID="41c61425dff61151f6aaa526b4c2e26f623ef56d26db2a80b8ad47e6a50b7ffc" exitCode=0 Oct 31 01:00:25.855631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.855174 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" event={"ID":"ed858859-aac8-4b37-9581-eb23c59b9ec6","Type":"ContainerDied","Data":"41c61425dff61151f6aaa526b4c2e26f623ef56d26db2a80b8ad47e6a50b7ffc"} Oct 31 01:00:25.856869 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.856839 5978 generic.go:356] "Generic (PLEG): container finished" podID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerID="131b4c353a547a5c37f72b3cf83aedf909dd243500aed904aeedadaa68e2edfe" exitCode=0 Oct 31 01:00:25.856986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.856900 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" event={"ID":"cc8883f9-eb27-4d37-8f96-1dc644f9ef13","Type":"ContainerDied","Data":"131b4c353a547a5c37f72b3cf83aedf909dd243500aed904aeedadaa68e2edfe"} Oct 31 01:00:25.885468 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.885433 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 01:00:25.993310 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.993271 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-envoy\") pod \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " Oct 31 01:00:25.993509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.993416 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-socket\") pod \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " Oct 31 01:00:25.993509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.993479 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-kube-api-access-rsxst\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-kube-api-access-rsxst\") pod \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " Oct 31 01:00:25.993653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.993532 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-token\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-token\") pod \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " Oct 31 01:00:25.993723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.993701 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-podinfo\") pod \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " Oct 31 01:00:25.993870 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.993841 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-socket" pod "cc8883f9-eb27-4d37-8f96-1dc644f9ef13" (UID: "cc8883f9-eb27-4d37-8f96-1dc644f9ef13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:25.993952 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.993863 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-data\") pod \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " Oct 31 01:00:25.994015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.993982 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istiod-ca-cert\") pod \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " Oct 31 01:00:25.994105 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.994086 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-certs\") pod \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " Oct 31 01:00:25.994171 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.994093 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-data" pod "cc8883f9-eb27-4d37-8f96-1dc644f9ef13" (UID: "cc8883f9-eb27-4d37-8f96-1dc644f9ef13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:25.994171 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.994158 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-credential-socket\") pod \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\" (UID: \"cc8883f9-eb27-4d37-8f96-1dc644f9ef13\") " Oct 31 01:00:25.994360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.994337 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-certs" pod "cc8883f9-eb27-4d37-8f96-1dc644f9ef13" (UID: "cc8883f9-eb27-4d37-8f96-1dc644f9ef13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:25.994733 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.994708 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istiod-ca-cert" pod "cc8883f9-eb27-4d37-8f96-1dc644f9ef13" (UID: "cc8883f9-eb27-4d37-8f96-1dc644f9ef13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:25.994971 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.994949 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-credential-socket" pod "cc8883f9-eb27-4d37-8f96-1dc644f9ef13" (UID: "cc8883f9-eb27-4d37-8f96-1dc644f9ef13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:25.995229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.995202 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:25.995312 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.995235 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:25.995312 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.995248 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:25.995312 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.995264 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:25.995312 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.995279 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:25.996534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.996508 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-kube-api-access-rsxst" pod "cc8883f9-eb27-4d37-8f96-1dc644f9ef13" (UID: "cc8883f9-eb27-4d37-8f96-1dc644f9ef13"). InnerVolumeSpecName "kube-api-access-rsxst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:25.997552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.997531 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-envoy" pod "cc8883f9-eb27-4d37-8f96-1dc644f9ef13" (UID: "cc8883f9-eb27-4d37-8f96-1dc644f9ef13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:25.997728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.997701 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-token" pod "cc8883f9-eb27-4d37-8f96-1dc644f9ef13" (UID: "cc8883f9-eb27-4d37-8f96-1dc644f9ef13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:25.997855 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:25.997836 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-podinfo" pod "cc8883f9-eb27-4d37-8f96-1dc644f9ef13" (UID: "cc8883f9-eb27-4d37-8f96-1dc644f9ef13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:26.096772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.096707 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.096772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.096739 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxst\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-kube-api-access-rsxst\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.096772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.096752 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.096772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.096761 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8883f9-eb27-4d37-8f96-1dc644f9ef13-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.297465 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.297421 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 01:00:26.299481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299436 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-envoy\") pod \"de28387c-28f7-484b-b722-237c98a33d1b\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " Oct 31 01:00:26.299624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299498 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-socket\") pod \"de28387c-28f7-484b-b722-237c98a33d1b\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " Oct 31 01:00:26.299624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299537 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-kube-api-access-r659g\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-kube-api-access-r659g\") pod \"de28387c-28f7-484b-b722-237c98a33d1b\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " Oct 31 01:00:26.299624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299567 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de28387c-28f7-484b-b722-237c98a33d1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de28387c-28f7-484b-b722-237c98a33d1b-istio-podinfo\") pod \"de28387c-28f7-484b-b722-237c98a33d1b\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " Oct 31 01:00:26.299771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299641 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-credential-socket\") pod \"de28387c-28f7-484b-b722-237c98a33d1b\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " Oct 31 01:00:26.299771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299694 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-data\") pod \"de28387c-28f7-484b-b722-237c98a33d1b\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " Oct 31 01:00:26.299960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299777 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-istio-token\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-istio-token\") pod \"de28387c-28f7-484b-b722-237c98a33d1b\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " Oct 31 01:00:26.299960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299826 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de28387c-28f7-484b-b722-237c98a33d1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de28387c-28f7-484b-b722-237c98a33d1b-istiod-ca-cert\") pod \"de28387c-28f7-484b-b722-237c98a33d1b\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " Oct 31 01:00:26.299960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299860 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-socket" pod "de28387c-28f7-484b-b722-237c98a33d1b" (UID: "de28387c-28f7-484b-b722-237c98a33d1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:26.299960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299913 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-certs\") pod \"de28387c-28f7-484b-b722-237c98a33d1b\" (UID: \"de28387c-28f7-484b-b722-237c98a33d1b\") " Oct 31 01:00:26.300166 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.299981 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-credential-socket" pod "de28387c-28f7-484b-b722-237c98a33d1b" (UID: "de28387c-28f7-484b-b722-237c98a33d1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:26.300166 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.300019 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-data" pod "de28387c-28f7-484b-b722-237c98a33d1b" (UID: "de28387c-28f7-484b-b722-237c98a33d1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:26.300341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.300318 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de28387c-28f7-484b-b722-237c98a33d1b-istiod-ca-cert" pod "de28387c-28f7-484b-b722-237c98a33d1b" (UID: "de28387c-28f7-484b-b722-237c98a33d1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:26.300341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.300328 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-certs" pod "de28387c-28f7-484b-b722-237c98a33d1b" (UID: "de28387c-28f7-484b-b722-237c98a33d1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:26.300669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.300651 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.300743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.300671 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.300743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.300682 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.300743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.300690 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de28387c-28f7-484b-b722-237c98a33d1b-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.300743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.300698 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.301830 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.301812 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de28387c-28f7-484b-b722-237c98a33d1b-istio-podinfo" pod "de28387c-28f7-484b-b722-237c98a33d1b" (UID: "de28387c-28f7-484b-b722-237c98a33d1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:26.302144 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.302126 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-istio-token" pod "de28387c-28f7-484b-b722-237c98a33d1b" (UID: "de28387c-28f7-484b-b722-237c98a33d1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:26.302222 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.302154 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-envoy" pod "de28387c-28f7-484b-b722-237c98a33d1b" (UID: "de28387c-28f7-484b-b722-237c98a33d1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:26.302222 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.302158 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-kube-api-access-r659g" pod "de28387c-28f7-484b-b722-237c98a33d1b" (UID: "de28387c-28f7-484b-b722-237c98a33d1b"). InnerVolumeSpecName "kube-api-access-r659g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:26.386028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.385916 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:26.403598 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.403563 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.403598 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.403599 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de28387c-28f7-484b-b722-237c98a33d1b-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.403805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.403614 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r659g\" (UniqueName: \"kubernetes.io/projected/de28387c-28f7-484b-b722-237c98a33d1b-kube-api-access-r659g\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.403805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.403629 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de28387c-28f7-484b-b722-237c98a33d1b-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:26.485727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.485688 5978 patch_prober.go:29] interesting pod/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:00:26.485879 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.485749 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:00:26.867667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.867631 5978 generic.go:356] "Generic (PLEG): container finished" podID="de28387c-28f7-484b-b722-237c98a33d1b" containerID="32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca" exitCode=0 Oct 31 01:00:26.868144 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.867692 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" Oct 31 01:00:26.868144 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.867710 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" event={"ID":"de28387c-28f7-484b-b722-237c98a33d1b","Type":"ContainerDied","Data":"32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca"} Oct 31 01:00:26.868144 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.867774 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l" event={"ID":"de28387c-28f7-484b-b722-237c98a33d1b","Type":"ContainerDied","Data":"366b4695b33c18d85b86442fb1944944e561372982bdff18c71de5ab61e24844"} Oct 31 01:00:26.868144 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.867804 5978 scope.go:122] "RemoveContainer" containerID="32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca" Oct 31 01:00:26.871376 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.869877 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" event={"ID":"cc8883f9-eb27-4d37-8f96-1dc644f9ef13","Type":"ContainerDied","Data":"23e7d2afe538794a1696af3dd66aee309ec3fd0efd00ee048339bd1edab4805f"} Oct 31 01:00:26.871376 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.869893 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847" Oct 31 01:00:26.882437 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.882410 5978 scope.go:122] "RemoveContainer" containerID="32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca" Oct 31 01:00:26.882919 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:26.882892 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca\": container with ID starting with 32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca not found: ID does not exist" containerID="32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca" Oct 31 01:00:26.883058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.882927 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca"} err="failed to get container status \"32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca\": rpc error: code = NotFound desc = could not find container \"32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca\": container with ID starting with 32c79b02d7c886af35e0bff6612e37bedb1586c2b3362bd1eccb1583206ed0ca not found: ID does not exist" Oct 31 01:00:26.883058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.882954 5978 scope.go:122] "RemoveContainer" containerID="131b4c353a547a5c37f72b3cf83aedf909dd243500aed904aeedadaa68e2edfe" Oct 31 01:00:26.918160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.918124 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l"] Oct 31 01:00:26.928000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.927963 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-735123-openshift-default-5889966f5-55w4l"] Oct 31 01:00:26.963708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.961888 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847"] Oct 31 01:00:26.963923 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:26.963886 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-db847"] Oct 31 01:00:27.059050 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.059025 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:27.115359 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.115320 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed858859-aac8-4b37-9581-eb23c59b9ec6-kube-api-access-swk24\" (UniqueName: \"kubernetes.io/projected/ed858859-aac8-4b37-9581-eb23c59b9ec6-kube-api-access-swk24\") pod \"ed858859-aac8-4b37-9581-eb23c59b9ec6\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " Oct 31 01:00:27.115841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.115809 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-openshift-ca\") pod \"ed858859-aac8-4b37-9581-eb23c59b9ec6\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " Oct 31 01:00:27.127615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.117251 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-openshift-ca" pod "ed858859-aac8-4b37-9581-eb23c59b9ec6" (UID: "ed858859-aac8-4b37-9581-eb23c59b9ec6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:27.129739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.129678 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed858859-aac8-4b37-9581-eb23c59b9ec6-kube-api-access-swk24" pod "ed858859-aac8-4b37-9581-eb23c59b9ec6" (UID: "ed858859-aac8-4b37-9581-eb23c59b9ec6"). InnerVolumeSpecName "kube-api-access-swk24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:27.217894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.217857 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-xks-ca\") pod \"ed858859-aac8-4b37-9581-eb23c59b9ec6\" (UID: \"ed858859-aac8-4b37-9581-eb23c59b9ec6\") " Oct 31 01:00:27.218738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.218717 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swk24\" (UniqueName: \"kubernetes.io/projected/ed858859-aac8-4b37-9581-eb23c59b9ec6-kube-api-access-swk24\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.218845 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.218740 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.219845 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.219818 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-xks-ca" pod "ed858859-aac8-4b37-9581-eb23c59b9ec6" (UID: "ed858859-aac8-4b37-9581-eb23c59b9ec6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:27.320923 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.320888 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed858859-aac8-4b37-9581-eb23c59b9ec6-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.406749 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.406683 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.422845 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-data\") pod \"aab1607a-7d55-418b-9ff7-725222d81510\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.422965 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-kube-api-access-v29gj\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-kube-api-access-v29gj\") pod \"aab1607a-7d55-418b-9ff7-725222d81510\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.423062 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-certs\") pod \"aab1607a-7d55-418b-9ff7-725222d81510\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.423244 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-socket\") pod \"aab1607a-7d55-418b-9ff7-725222d81510\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.423413 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-istio-token\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-istio-token\") pod \"aab1607a-7d55-418b-9ff7-725222d81510\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.423484 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aab1607a-7d55-418b-9ff7-725222d81510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab1607a-7d55-418b-9ff7-725222d81510-istio-podinfo\") pod \"aab1607a-7d55-418b-9ff7-725222d81510\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.423545 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-envoy\") pod \"aab1607a-7d55-418b-9ff7-725222d81510\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.423580 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-credential-socket\") pod \"aab1607a-7d55-418b-9ff7-725222d81510\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.423648 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab1607a-7d55-418b-9ff7-725222d81510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab1607a-7d55-418b-9ff7-725222d81510-istiod-ca-cert\") pod \"aab1607a-7d55-418b-9ff7-725222d81510\" (UID: \"aab1607a-7d55-418b-9ff7-725222d81510\") " Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.425511 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab1607a-7d55-418b-9ff7-725222d81510-istiod-ca-cert" pod "aab1607a-7d55-418b-9ff7-725222d81510" (UID: "aab1607a-7d55-418b-9ff7-725222d81510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.425622 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-socket" pod "aab1607a-7d55-418b-9ff7-725222d81510" (UID: "aab1607a-7d55-418b-9ff7-725222d81510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.425784 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-data" pod "aab1607a-7d55-418b-9ff7-725222d81510" (UID: "aab1607a-7d55-418b-9ff7-725222d81510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.425863 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-certs" pod "aab1607a-7d55-418b-9ff7-725222d81510" (UID: "aab1607a-7d55-418b-9ff7-725222d81510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.426045 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-credential-socket" pod "aab1607a-7d55-418b-9ff7-725222d81510" (UID: "aab1607a-7d55-418b-9ff7-725222d81510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.433117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.430881 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-istio-token" pod "aab1607a-7d55-418b-9ff7-725222d81510" (UID: "aab1607a-7d55-418b-9ff7-725222d81510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:27.443473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.436524 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aab1607a-7d55-418b-9ff7-725222d81510-istio-podinfo" pod "aab1607a-7d55-418b-9ff7-725222d81510" (UID: "aab1607a-7d55-418b-9ff7-725222d81510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:27.443473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.438739 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-envoy" pod "aab1607a-7d55-418b-9ff7-725222d81510" (UID: "aab1607a-7d55-418b-9ff7-725222d81510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.446512 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.444637 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-kube-api-access-v29gj" pod "aab1607a-7d55-418b-9ff7-725222d81510" (UID: "aab1607a-7d55-418b-9ff7-725222d81510"). InnerVolumeSpecName "kube-api-access-v29gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:27.511228 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.511201 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 01:00:27.525361 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.525325 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-certs\") pod \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " Oct 31 01:00:27.525602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.525384 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-envoy\") pod \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " Oct 31 01:00:27.525602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.525431 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-podinfo\") pod \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " Oct 31 01:00:27.525722 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.525622 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-data\") pod \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " Oct 31 01:00:27.525722 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.525687 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istiod-ca-cert\") pod \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " Oct 31 01:00:27.525827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.525772 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-certs" pod "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" (UID: "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.525934 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.525915 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-token\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-token\") pod \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " Oct 31 01:00:27.526008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.525993 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-credential-socket\") pod \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " Oct 31 01:00:27.526121 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.526101 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-socket\") pod \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " Oct 31 01:00:27.526224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.526206 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-kube-api-access-hhs6q\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-kube-api-access-hhs6q\") pod \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\" (UID: \"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8\") " Oct 31 01:00:27.526725 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.526698 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-data" pod "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" (UID: "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.527287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.527259 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istiod-ca-cert" pod "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" (UID: "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:27.527580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.527556 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab1607a-7d55-418b-9ff7-725222d81510-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.527732 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.527716 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.527935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.527850 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.528110 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.528094 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab1607a-7d55-418b-9ff7-725222d81510-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.528225 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.528214 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.528341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.528330 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.528440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.528428 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v29gj\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-kube-api-access-v29gj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.528561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.528549 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.528660 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.528639 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab1607a-7d55-418b-9ff7-725222d81510-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.528748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.528658 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.528748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.528671 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab1607a-7d55-418b-9ff7-725222d81510-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.528748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.527619 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-socket" pod "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" (UID: "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.530129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.530103 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-credential-socket" pod "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" (UID: "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.532913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.532891 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-token" pod "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" (UID: "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:27.533033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.532922 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-kube-api-access-hhs6q" pod "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" (UID: "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8"). InnerVolumeSpecName "kube-api-access-hhs6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:27.534337 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.534313 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-podinfo" pod "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" (UID: "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:27.534570 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.534542 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-envoy" pod "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" (UID: "e2a8cfe0-2892-4b0c-9fab-aa158abdcab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:27.630044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.630007 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.630044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.630040 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.630044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.630049 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.630288 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.630061 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.630288 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.630071 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhs6q\" (UniqueName: \"kubernetes.io/projected/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-kube-api-access-hhs6q\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.630288 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.630079 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.630288 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.630087 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:27.882336 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.882300 5978 generic.go:356] "Generic (PLEG): container finished" podID="aab1607a-7d55-418b-9ff7-725222d81510" containerID="ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4" exitCode=0 Oct 31 01:00:27.882884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.882381 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" Oct 31 01:00:27.882884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.882382 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" event={"ID":"aab1607a-7d55-418b-9ff7-725222d81510","Type":"ContainerDied","Data":"ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4"} Oct 31 01:00:27.882884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.882528 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz" event={"ID":"aab1607a-7d55-418b-9ff7-725222d81510","Type":"ContainerDied","Data":"b055b1492147f6610af7245b1aa0ab957b97522a49bd9dbc1d5a96f9a1396dc7"} Oct 31 01:00:27.882884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.882571 5978 scope.go:122] "RemoveContainer" containerID="ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4" Oct 31 01:00:27.884070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.883999 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" event={"ID":"ed858859-aac8-4b37-9581-eb23c59b9ec6","Type":"ContainerDied","Data":"924489e645fe66952823dfe032511c75955ca2bca8666bcc03542bcc3d837c03"} Oct 31 01:00:27.884070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.884041 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="924489e645fe66952823dfe032511c75955ca2bca8666bcc03542bcc3d837c03" Oct 31 01:00:27.884180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.884087 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk" Oct 31 01:00:27.888273 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.888246 5978 generic.go:356] "Generic (PLEG): container finished" podID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerID="2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162" exitCode=0 Oct 31 01:00:27.888442 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.888300 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" Oct 31 01:00:27.888442 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.888372 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" event={"ID":"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8","Type":"ContainerDied","Data":"2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162"} Oct 31 01:00:27.888442 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.888412 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm" event={"ID":"e2a8cfe0-2892-4b0c-9fab-aa158abdcab8","Type":"ContainerDied","Data":"e53868f2a05fc59b563c985e93b6936a0b48b1e96769b10257b29d011f5de02c"} Oct 31 01:00:27.897357 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.897336 5978 scope.go:122] "RemoveContainer" containerID="ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4" Oct 31 01:00:27.897750 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:27.897726 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4\": container with ID starting with ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4 not found: ID does not exist" containerID="ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4" Oct 31 01:00:27.897837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.897763 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4"} err="failed to get container status \"ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4\": rpc error: code = NotFound desc = could not find container \"ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4\": container with ID starting with ff35585ed2f0792ab419ae4e625f472c6e829d10007420937dd46f4ddce436d4 not found: ID does not exist" Oct 31 01:00:27.897837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.897786 5978 scope.go:122] "RemoveContainer" containerID="2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162" Oct 31 01:00:27.910654 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.910633 5978 scope.go:122] "RemoveContainer" containerID="2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162" Oct 31 01:00:27.910989 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:27.910970 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162\": container with ID starting with 2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162 not found: ID does not exist" containerID="2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162" Oct 31 01:00:27.911052 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:27.911003 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162"} err="failed to get container status \"2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162\": rpc error: code = NotFound desc = could not find container \"2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162\": container with ID starting with 2b55d76314627d69e6b2902b43c14d2c4f2a55444239353d28fe1be93bb31162 not found: ID does not exist" Oct 31 01:00:28.061289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:28.042511 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm"] Oct 31 01:00:28.082303 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:28.082264 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-wmmjm"] Oct 31 01:00:28.200011 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:28.199925 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz"] Oct 31 01:00:28.205841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:28.205810 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-7rfqz"] Oct 31 01:00:28.300918 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:28.300881 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:28.429364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:28.429329 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab1607a-7d55-418b-9ff7-725222d81510" path="/var/lib/kubelet/pods/aab1607a-7d55-418b-9ff7-725222d81510/volumes" Oct 31 01:00:28.429966 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:28.429945 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" path="/var/lib/kubelet/pods/cc8883f9-eb27-4d37-8f96-1dc644f9ef13/volumes" Oct 31 01:00:28.430480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:28.430447 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de28387c-28f7-484b-b722-237c98a33d1b" path="/var/lib/kubelet/pods/de28387c-28f7-484b-b722-237c98a33d1b/volumes" Oct 31 01:00:28.431025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:28.431010 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" path="/var/lib/kubelet/pods/e2a8cfe0-2892-4b0c-9fab-aa158abdcab8/volumes" Oct 31 01:00:29.947592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:29.946928 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:31.553998 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:31.553950 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:33.184660 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:00:33.184614 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:00:33.269561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:33.269522 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:34.186086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:34.186039 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:34.329548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:34.329512 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:34.939943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:34.939902 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:36.553088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:36.553054 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:38.309803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:38.309770 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:40.067593 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.067558 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:40.229411 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.229376 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:40.532334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.532250 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj"] Oct 31 01:00:40.533076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533061 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" Oct 31 01:00:40.533117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533078 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" Oct 31 01:00:40.533117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533098 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ed858859-aac8-4b37-9581-eb23c59b9ec6" containerName="revoke-keys" Oct 31 01:00:40.533117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533104 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed858859-aac8-4b37-9581-eb23c59b9ec6" containerName="revoke-keys" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533135 5978 cpu_manager.go:414] "containerMap: removing container" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533142 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533151 5978 cpu_manager.go:414] "containerMap: removing container" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533156 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533178 5978 cpu_manager.go:414] "containerMap: removing container" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533183 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533198 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" containerName="cleanup" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533202 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca573556-53d4-431a-8f75-1637ebaa618b" containerName="cleanup" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533212 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerName="cleanup" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533217 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcf361c-5cc7-483e-93f2-063c1e38e0df" containerName="cleanup" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533228 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerName="cleanup" Oct 31 01:00:40.533277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533233 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a3049f-1270-4d3c-a8ad-f13ffe40a214" containerName="cleanup" Oct 31 01:00:40.533702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533335 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="380d9685-5509-41ec-85ea-f4488a8e1461" containerName="cleanup" Oct 31 01:00:40.533702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533346 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8883f9-eb27-4d37-8f96-1dc644f9ef13" containerName="istio-proxy" Oct 31 01:00:40.533702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533353 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c884e066-0089-4a02-a980-6bcf9b5c8e89" containerName="cleanup" Oct 31 01:00:40.533702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533359 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab1607a-7d55-418b-9ff7-725222d81510" containerName="istio-proxy" Oct 31 01:00:40.533702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533368 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="de28387c-28f7-484b-b722-237c98a33d1b" containerName="istio-proxy" Oct 31 01:00:40.533702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533377 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14a9d7a-f8a3-43f5-a507-cdd72fa85df0" containerName="cleanup" Oct 31 01:00:40.533702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533386 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a8cfe0-2892-4b0c-9fab-aa158abdcab8" containerName="istio-proxy" Oct 31 01:00:40.533702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.533393 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed858859-aac8-4b37-9581-eb23c59b9ec6" containerName="revoke-keys" Oct 31 01:00:40.537341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.537319 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.558838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.558796 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj"] Oct 31 01:00:40.563423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.563394 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:00:40.563611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.563473 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:00:40.613116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.613080 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tfg\" (UniqueName: \"kubernetes.io/projected/cdc0281b-df0d-4669-b967-b75e964e1ce4-kube-api-access-28tfg\") pod \"maas-api-revoke-sub-8706aaa1-lrwcj\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.613301 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.613148 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8706aaa1-lrwcj\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.613393 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.613365 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8706aaa1-lrwcj\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.714774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.714738 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tfg\" (UniqueName: \"kubernetes.io/projected/cdc0281b-df0d-4669-b967-b75e964e1ce4-kube-api-access-28tfg\") pod \"maas-api-revoke-sub-8706aaa1-lrwcj\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.714968 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.714822 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8706aaa1-lrwcj\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.715026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.715008 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8706aaa1-lrwcj\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.715585 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.715563 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8706aaa1-lrwcj\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.717544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.717524 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8706aaa1-lrwcj\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.736746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.736717 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tfg\" (UniqueName: \"kubernetes.io/projected/cdc0281b-df0d-4669-b967-b75e964e1ce4-kube-api-access-28tfg\") pod \"maas-api-revoke-sub-8706aaa1-lrwcj\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:40.852522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:40.852489 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:41.159067 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:41.159003 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj"] Oct 31 01:00:41.160475 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:00:41.160426 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc0281b_df0d_4669_b967_b75e964e1ce4.slice/crio-3257adeab80c0a3f30eadb5e58384e3f4de292b1083bfa42a7dce14e80e2ac25 WatchSource:0}: Error finding container 3257adeab80c0a3f30eadb5e58384e3f4de292b1083bfa42a7dce14e80e2ac25: Status 404 returned error can't find the container with id 3257adeab80c0a3f30eadb5e58384e3f4de292b1083bfa42a7dce14e80e2ac25 Oct 31 01:00:41.914477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:41.907701 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:42.091997 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:42.091946 5978 generic.go:356] "Generic (PLEG): container finished" podID="cdc0281b-df0d-4669-b967-b75e964e1ce4" containerID="43b9b6b0499d64e478ed1da4aa40829c592a9b95a64212d19faddb83edeef69f" exitCode=0 Oct 31 01:00:42.092205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:42.092013 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" event={"ID":"cdc0281b-df0d-4669-b967-b75e964e1ce4","Type":"ContainerDied","Data":"43b9b6b0499d64e478ed1da4aa40829c592a9b95a64212d19faddb83edeef69f"} Oct 31 01:00:42.092205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:42.092099 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" event={"ID":"cdc0281b-df0d-4669-b967-b75e964e1ce4","Type":"ContainerStarted","Data":"3257adeab80c0a3f30eadb5e58384e3f4de292b1083bfa42a7dce14e80e2ac25"} Oct 31 01:00:43.291834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.291802 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:43.357925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.357767 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-openshift-ca\") pod \"cdc0281b-df0d-4669-b967-b75e964e1ce4\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " Oct 31 01:00:43.358183 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.357991 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc0281b-df0d-4669-b967-b75e964e1ce4-kube-api-access-28tfg\" (UniqueName: \"kubernetes.io/projected/cdc0281b-df0d-4669-b967-b75e964e1ce4-kube-api-access-28tfg\") pod \"cdc0281b-df0d-4669-b967-b75e964e1ce4\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " Oct 31 01:00:43.358183 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.358077 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-xks-ca\") pod \"cdc0281b-df0d-4669-b967-b75e964e1ce4\" (UID: \"cdc0281b-df0d-4669-b967-b75e964e1ce4\") " Oct 31 01:00:43.359558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.359527 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-openshift-ca" pod "cdc0281b-df0d-4669-b967-b75e964e1ce4" (UID: "cdc0281b-df0d-4669-b967-b75e964e1ce4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:43.361578 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.361424 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-xks-ca" pod "cdc0281b-df0d-4669-b967-b75e964e1ce4" (UID: "cdc0281b-df0d-4669-b967-b75e964e1ce4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:43.362250 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.362219 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc0281b-df0d-4669-b967-b75e964e1ce4-kube-api-access-28tfg" pod "cdc0281b-df0d-4669-b967-b75e964e1ce4" (UID: "cdc0281b-df0d-4669-b967-b75e964e1ce4"). InnerVolumeSpecName "kube-api-access-28tfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:43.453611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.453572 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:43.460216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.460186 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:43.460216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.460218 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0281b-df0d-4669-b967-b75e964e1ce4-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:43.460395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.460232 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28tfg\" (UniqueName: \"kubernetes.io/projected/cdc0281b-df0d-4669-b967-b75e964e1ce4-kube-api-access-28tfg\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:43.697427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:43.697317 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:44.109711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:44.109679 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" Oct 31 01:00:44.109897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:44.109707 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj" event={"ID":"cdc0281b-df0d-4669-b967-b75e964e1ce4","Type":"ContainerDied","Data":"3257adeab80c0a3f30eadb5e58384e3f4de292b1083bfa42a7dce14e80e2ac25"} Oct 31 01:00:44.109897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:44.109762 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3257adeab80c0a3f30eadb5e58384e3f4de292b1083bfa42a7dce14e80e2ac25" Oct 31 01:00:44.954594 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:44.952683 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4"] Oct 31 01:00:44.954594 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:44.952954 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" podUID="639a34f2-fa2a-42c6-9c2f-cdf2664495ee" containerName="maas-api" containerID="cri-o://8aa4045ea75387fc50dbacf296cf9bcf4f83152bf753308429a681856f709311" gracePeriod=30 Oct 31 01:00:45.057984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.057946 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:45.120186 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.120151 5978 generic.go:356] "Generic (PLEG): container finished" podID="639a34f2-fa2a-42c6-9c2f-cdf2664495ee" containerID="8aa4045ea75387fc50dbacf296cf9bcf4f83152bf753308429a681856f709311" exitCode=0 Oct 31 01:00:45.120395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.120224 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" event={"ID":"639a34f2-fa2a-42c6-9c2f-cdf2664495ee","Type":"ContainerDied","Data":"8aa4045ea75387fc50dbacf296cf9bcf4f83152bf753308429a681856f709311"} Oct 31 01:00:45.488417 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.482717 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 01:00:45.656595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.656180 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-openshift-service-ca\") pod \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " Oct 31 01:00:45.656595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.656529 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-metrics-certs\") pod \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " Oct 31 01:00:45.656856 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.656605 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-maas-api-tls\") pod \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " Oct 31 01:00:45.656856 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.656720 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-kube-api-access-cxkwz\" (UniqueName: \"kubernetes.io/projected/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-kube-api-access-cxkwz\") pod \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\" (UID: \"639a34f2-fa2a-42c6-9c2f-cdf2664495ee\") " Oct 31 01:00:45.658076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.658045 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-openshift-service-ca" pod "639a34f2-fa2a-42c6-9c2f-cdf2664495ee" (UID: "639a34f2-fa2a-42c6-9c2f-cdf2664495ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:45.660395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.660363 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-kube-api-access-cxkwz" pod "639a34f2-fa2a-42c6-9c2f-cdf2664495ee" (UID: "639a34f2-fa2a-42c6-9c2f-cdf2664495ee"). InnerVolumeSpecName "kube-api-access-cxkwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:45.660766 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.660740 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-metrics-certs" pod "639a34f2-fa2a-42c6-9c2f-cdf2664495ee" (UID: "639a34f2-fa2a-42c6-9c2f-cdf2664495ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:45.667771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.667735 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-maas-api-tls" pod "639a34f2-fa2a-42c6-9c2f-cdf2664495ee" (UID: "639a34f2-fa2a-42c6-9c2f-cdf2664495ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:45.758987 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.758949 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:45.758987 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.758981 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:45.758987 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.758990 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxkwz\" (UniqueName: \"kubernetes.io/projected/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-kube-api-access-cxkwz\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:45.759237 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.759001 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/639a34f2-fa2a-42c6-9c2f-cdf2664495ee-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:45.812240 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.812205 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:45.972557 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:45.970741 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:46.037756 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.037720 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk"] Oct 31 01:00:46.045968 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.045933 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d5621-vf8pk"] Oct 31 01:00:46.131741 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.131706 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" Oct 31 01:00:46.131954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.131744 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4" event={"ID":"639a34f2-fa2a-42c6-9c2f-cdf2664495ee","Type":"ContainerDied","Data":"e430d2ef7028d3f72aa325dec697ea534dc147c65f8aeae718ab95224d23b8d2"} Oct 31 01:00:46.131954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.131812 5978 scope.go:122] "RemoveContainer" containerID="8aa4045ea75387fc50dbacf296cf9bcf4f83152bf753308429a681856f709311" Oct 31 01:00:46.178522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.171893 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:46.209720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.209683 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4"] Oct 31 01:00:46.220901 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.220866 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d5621-d4bdcb578-659t4"] Oct 31 01:00:46.474475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.471618 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639a34f2-fa2a-42c6-9c2f-cdf2664495ee" path="/var/lib/kubelet/pods/639a34f2-fa2a-42c6-9c2f-cdf2664495ee/volumes" Oct 31 01:00:46.474475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.472669 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed858859-aac8-4b37-9581-eb23c59b9ec6" path="/var/lib/kubelet/pods/ed858859-aac8-4b37-9581-eb23c59b9ec6/volumes" Oct 31 01:00:46.567534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:46.560171 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:47.544478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:47.539595 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:48.344477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:48.333604 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:50.001846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:50.001808 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:51.893330 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:51.893286 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:53.169887 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:53.169849 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:53.321058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:53.321008 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:53.539948 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:53.539857 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:53.709715 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:53.709677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:54.029503 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.029465 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k"] Oct 31 01:00:54.030790 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.030766 5978 cpu_manager.go:414] "containerMap: removing container" podUID="cdc0281b-df0d-4669-b967-b75e964e1ce4" containerName="revoke-keys" Oct 31 01:00:54.030857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.030792 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc0281b-df0d-4669-b967-b75e964e1ce4" containerName="revoke-keys" Oct 31 01:00:54.030857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.030810 5978 cpu_manager.go:414] "containerMap: removing container" podUID="639a34f2-fa2a-42c6-9c2f-cdf2664495ee" containerName="maas-api" Oct 31 01:00:54.030857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.030817 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639a34f2-fa2a-42c6-9c2f-cdf2664495ee" containerName="maas-api" Oct 31 01:00:54.030987 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.030975 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc0281b-df0d-4669-b967-b75e964e1ce4" containerName="revoke-keys" Oct 31 01:00:54.031029 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.030994 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="639a34f2-fa2a-42c6-9c2f-cdf2664495ee" containerName="maas-api" Oct 31 01:00:54.038034 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.038008 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.053632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.053601 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k"] Oct 31 01:00:54.071879 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.071846 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:00:54.071879 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.071845 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:00:54.204404 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.204368 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.204879 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.204538 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.204879 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.204603 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm75s\" (UniqueName: \"kubernetes.io/projected/2f8b781a-5863-4aae-a186-e5a03aea7914-kube-api-access-nm75s\") pod \"maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.306932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.306849 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.307139 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.306943 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.307139 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.306975 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm75s\" (UniqueName: \"kubernetes.io/projected/2f8b781a-5863-4aae-a186-e5a03aea7914-kube-api-access-nm75s\") pod \"maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.307854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.307813 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.309622 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.309596 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.330080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.330048 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm75s\" (UniqueName: \"kubernetes.io/projected/2f8b781a-5863-4aae-a186-e5a03aea7914-kube-api-access-nm75s\") pod \"maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.357076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.357039 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:54.683418 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:54.683389 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k"] Oct 31 01:00:54.686669 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:00:54.686638 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8b781a_5863_4aae_a186_e5a03aea7914.slice/crio-25c2cd87db9ad38b3aec3586f8dbe4c7a175980b5195ab30d6a726567ad26c3f WatchSource:0}: Error finding container 25c2cd87db9ad38b3aec3586f8dbe4c7a175980b5195ab30d6a726567ad26c3f: Status 404 returned error can't find the container with id 25c2cd87db9ad38b3aec3586f8dbe4c7a175980b5195ab30d6a726567ad26c3f Oct 31 01:00:55.236706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:55.236616 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" event={"ID":"2f8b781a-5863-4aae-a186-e5a03aea7914","Type":"ContainerStarted","Data":"cfdbd0abb2a561899de3922e23a4fec6ed9049a797048900b5d898467be545fe"} Oct 31 01:00:55.236706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:55.236679 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" event={"ID":"2f8b781a-5863-4aae-a186-e5a03aea7914","Type":"ContainerStarted","Data":"25c2cd87db9ad38b3aec3586f8dbe4c7a175980b5195ab30d6a726567ad26c3f"} Oct 31 01:00:55.274157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:55.274102 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" podStartSLOduration=1.274087534 podStartE2EDuration="1.274087534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:55.273070267 +0000 UTC m=+2472.971184045" watchObservedRunningTime="2026-10-31 01:00:55.274087534 +0000 UTC m=+2472.972201298" Oct 31 01:00:55.454744 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:55.454697 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:56.247793 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:56.247759 5978 generic.go:356] "Generic (PLEG): container finished" podID="2f8b781a-5863-4aae-a186-e5a03aea7914" containerID="cfdbd0abb2a561899de3922e23a4fec6ed9049a797048900b5d898467be545fe" exitCode=0 Oct 31 01:00:56.248229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:56.247850 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" event={"ID":"2f8b781a-5863-4aae-a186-e5a03aea7914","Type":"ContainerDied","Data":"cfdbd0abb2a561899de3922e23a4fec6ed9049a797048900b5d898467be545fe"} Oct 31 01:00:57.173141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.173059 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:57.271213 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.271172 5978 generic.go:356] "Generic (PLEG): container finished" podID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerID="1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d" exitCode=0 Oct 31 01:00:57.271767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.271242 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" event={"ID":"3068a265-cfd5-41b1-b2ed-ddac0d8553ab","Type":"ContainerDied","Data":"1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d"} Oct 31 01:00:57.271977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.271949 5978 scope.go:122] "RemoveContainer" containerID="1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d" Oct 31 01:00:57.275343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.275302 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:00:57.275491 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.275389 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:00:57.275491 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.275422 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:00:57.529583 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.529551 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:57.575389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.574929 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-openshift-ca\") pod \"2f8b781a-5863-4aae-a186-e5a03aea7914\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " Oct 31 01:00:57.575629 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.575351 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-openshift-ca" pod "2f8b781a-5863-4aae-a186-e5a03aea7914" (UID: "2f8b781a-5863-4aae-a186-e5a03aea7914"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:57.575629 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.575564 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8b781a-5863-4aae-a186-e5a03aea7914-kube-api-access-nm75s\" (UniqueName: \"kubernetes.io/projected/2f8b781a-5863-4aae-a186-e5a03aea7914-kube-api-access-nm75s\") pod \"2f8b781a-5863-4aae-a186-e5a03aea7914\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " Oct 31 01:00:57.575743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.575665 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-xks-ca\") pod \"2f8b781a-5863-4aae-a186-e5a03aea7914\" (UID: \"2f8b781a-5863-4aae-a186-e5a03aea7914\") " Oct 31 01:00:57.576608 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.576577 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:57.608750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.600049 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8b781a-5863-4aae-a186-e5a03aea7914-kube-api-access-nm75s" pod "2f8b781a-5863-4aae-a186-e5a03aea7914" (UID: "2f8b781a-5863-4aae-a186-e5a03aea7914"). InnerVolumeSpecName "kube-api-access-nm75s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:57.608750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.601556 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-xks-ca" pod "2f8b781a-5863-4aae-a186-e5a03aea7914" (UID: "2f8b781a-5863-4aae-a186-e5a03aea7914"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:57.677907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.677867 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm75s\" (UniqueName: \"kubernetes.io/projected/2f8b781a-5863-4aae-a186-e5a03aea7914-kube-api-access-nm75s\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:57.677907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:57.677909 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8b781a-5863-4aae-a186-e5a03aea7914-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:00:58.281224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:58.281197 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" Oct 31 01:00:58.281776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:58.281240 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k" event={"ID":"2f8b781a-5863-4aae-a186-e5a03aea7914","Type":"ContainerDied","Data":"25c2cd87db9ad38b3aec3586f8dbe4c7a175980b5195ab30d6a726567ad26c3f"} Oct 31 01:00:58.281776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:58.281304 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c2cd87db9ad38b3aec3586f8dbe4c7a175980b5195ab30d6a726567ad26c3f" Oct 31 01:00:58.283773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:58.283740 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" event={"ID":"3068a265-cfd5-41b1-b2ed-ddac0d8553ab","Type":"ContainerStarted","Data":"232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330"} Oct 31 01:00:58.992479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:58.989150 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:59.176208 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:59.176169 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 01:00:59.177532 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:59.177485 5978 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 01:00:59.177681 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:59.177538 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 01:00:59.409190 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:00:59.409155 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 01:01:00.177032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:00.176987 5978 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 01:01:00.177228 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:00.177063 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 01:01:00.576230 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:00.576180 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:01.076283 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.076245 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:01.182474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.180570 5978 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 01:01:01.182474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.180639 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 01:01:01.309685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.309651 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk"] Oct 31 01:01:01.310672 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.310641 5978 cpu_manager.go:414] "containerMap: removing container" podUID="2f8b781a-5863-4aae-a186-e5a03aea7914" containerName="revoke-keys" Oct 31 01:01:01.310672 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.310669 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8b781a-5863-4aae-a186-e5a03aea7914" containerName="revoke-keys" Oct 31 01:01:01.310923 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.310883 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8b781a-5863-4aae-a186-e5a03aea7914" containerName="revoke-keys" Oct 31 01:01:01.325519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.325425 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.344567 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.342058 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:01:01.344567 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.342863 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:01:01.351431 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.351397 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk"] Oct 31 01:01:01.383413 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.383377 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.383632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.383478 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.383780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.383760 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgbw5\" (UniqueName: \"kubernetes.io/projected/99a1f646-907c-4d5c-adbb-32dd752def1e-kube-api-access-pgbw5\") pod \"maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.485586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.485551 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgbw5\" (UniqueName: \"kubernetes.io/projected/99a1f646-907c-4d5c-adbb-32dd752def1e-kube-api-access-pgbw5\") pod \"maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.485810 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.485706 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.485810 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.485789 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.486549 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.486527 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.488214 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.488189 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.510121 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.510083 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgbw5\" (UniqueName: \"kubernetes.io/projected/99a1f646-907c-4d5c-adbb-32dd752def1e-kube-api-access-pgbw5\") pod \"maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:01.669071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:01.668984 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:02.045616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:02.045572 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk"] Oct 31 01:01:02.046263 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:01:02.046103 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a1f646_907c_4d5c_adbb_32dd752def1e.slice/crio-879b9a411f45d83cf80bc505036cfb9f97764b2f24d4689c3af8456e7f6c3f39 WatchSource:0}: Error finding container 879b9a411f45d83cf80bc505036cfb9f97764b2f24d4689c3af8456e7f6c3f39: Status 404 returned error can't find the container with id 879b9a411f45d83cf80bc505036cfb9f97764b2f24d4689c3af8456e7f6c3f39 Oct 31 01:01:02.177731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:02.177690 5978 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 01:01:02.177947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:02.177755 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 01:01:02.339392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:02.339351 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" event={"ID":"99a1f646-907c-4d5c-adbb-32dd752def1e","Type":"ContainerStarted","Data":"879b9a411f45d83cf80bc505036cfb9f97764b2f24d4689c3af8456e7f6c3f39"} Oct 31 01:01:02.453478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:02.452743 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:03.179028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:03.178987 5978 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 01:01:03.179440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:03.179076 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 01:01:03.184659 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:01:03.184617 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:01:03.349516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:03.349478 5978 generic.go:356] "Generic (PLEG): container finished" podID="99a1f646-907c-4d5c-adbb-32dd752def1e" containerID="80ff8de69a867e820aa42bd075e5ee464951f5b3d2e9f52f77b593e1c0810ee0" exitCode=0 Oct 31 01:01:03.349703 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:03.349540 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" event={"ID":"99a1f646-907c-4d5c-adbb-32dd752def1e","Type":"ContainerDied","Data":"80ff8de69a867e820aa42bd075e5ee464951f5b3d2e9f52f77b593e1c0810ee0"} Oct 31 01:01:04.181830 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.181800 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 01:01:04.183094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.183074 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 01:01:04.351305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.351268 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:04.567994 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.567967 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:04.658677 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.658639 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-openshift-ca\") pod \"99a1f646-907c-4d5c-adbb-32dd752def1e\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " Oct 31 01:01:04.658865 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.658793 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-xks-ca\") pod \"99a1f646-907c-4d5c-adbb-32dd752def1e\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " Oct 31 01:01:04.659033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.659008 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-openshift-ca" pod "99a1f646-907c-4d5c-adbb-32dd752def1e" (UID: "99a1f646-907c-4d5c-adbb-32dd752def1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:04.659069 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.659029 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a1f646-907c-4d5c-adbb-32dd752def1e-kube-api-access-pgbw5\" (UniqueName: \"kubernetes.io/projected/99a1f646-907c-4d5c-adbb-32dd752def1e-kube-api-access-pgbw5\") pod \"99a1f646-907c-4d5c-adbb-32dd752def1e\" (UID: \"99a1f646-907c-4d5c-adbb-32dd752def1e\") " Oct 31 01:01:04.659644 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.659620 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:04.660953 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.660930 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-xks-ca" pod "99a1f646-907c-4d5c-adbb-32dd752def1e" (UID: "99a1f646-907c-4d5c-adbb-32dd752def1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:04.661070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.661052 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a1f646-907c-4d5c-adbb-32dd752def1e-kube-api-access-pgbw5" pod "99a1f646-907c-4d5c-adbb-32dd752def1e" (UID: "99a1f646-907c-4d5c-adbb-32dd752def1e"). InnerVolumeSpecName "kube-api-access-pgbw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:04.761669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.761576 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a1f646-907c-4d5c-adbb-32dd752def1e-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:04.761669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.761619 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgbw5\" (UniqueName: \"kubernetes.io/projected/99a1f646-907c-4d5c-adbb-32dd752def1e-kube-api-access-pgbw5\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:04.809065 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.809021 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:04.968780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:04.968743 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:05.432486 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:05.432320 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" Oct 31 01:01:05.433022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:05.432576 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk" event={"ID":"99a1f646-907c-4d5c-adbb-32dd752def1e","Type":"ContainerDied","Data":"879b9a411f45d83cf80bc505036cfb9f97764b2f24d4689c3af8456e7f6c3f39"} Oct 31 01:01:05.433022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:05.432633 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="879b9a411f45d83cf80bc505036cfb9f97764b2f24d4689c3af8456e7f6c3f39" Oct 31 01:01:06.124048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:06.123997 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:07.753317 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:07.753125 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:09.453547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:09.451072 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:11.019563 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:11.019521 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:11.953905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:11.953865 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:12.110786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:12.110741 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:12.896949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:12.896908 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:14.585028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.584987 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:14.860116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.860027 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj"] Oct 31 01:01:14.861140 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.861112 5978 cpu_manager.go:414] "containerMap: removing container" podUID="99a1f646-907c-4d5c-adbb-32dd752def1e" containerName="revoke-keys" Oct 31 01:01:14.861140 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.861139 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a1f646-907c-4d5c-adbb-32dd752def1e" containerName="revoke-keys" Oct 31 01:01:14.861353 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.861339 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a1f646-907c-4d5c-adbb-32dd752def1e" containerName="revoke-keys" Oct 31 01:01:14.866080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.866051 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:14.889651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.889621 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj"] Oct 31 01:01:14.893438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.893406 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c942c9\"/\"kube-root-ca.crt\"" Oct 31 01:01:14.893438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.893406 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c942c9\"/\"openshift-service-ca.crt\"" Oct 31 01:01:14.893438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.893419 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c942c9\"/\"test-model-c942c9-kserve-self-signed-certs\"" Oct 31 01:01:14.893780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.893515 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c942c9\"/\"default-dockercfg-xr5kd\"" Oct 31 01:01:14.950518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.950447 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-tmp-dir\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:14.950720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.950552 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe0e284-ca30-495d-8462-c29e540c6bb7-tls-certs\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:14.950720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.950654 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-home\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:14.950720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.950707 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-model-cache\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:14.950892 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.950879 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-dshm\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:14.950931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.950908 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mzp\" (UniqueName: \"kubernetes.io/projected/5fe0e284-ca30-495d-8462-c29e540c6bb7-kube-api-access-h4mzp\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:14.970401 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:14.970359 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:15.052301 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.052268 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-tmp-dir\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.052517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.052314 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe0e284-ca30-495d-8462-c29e540c6bb7-tls-certs\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.052517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.052373 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-home\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.052517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.052419 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-model-cache\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.052735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.052598 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-dshm\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.052735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.052642 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mzp\" (UniqueName: \"kubernetes.io/projected/5fe0e284-ca30-495d-8462-c29e540c6bb7-kube-api-access-h4mzp\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.052735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.052692 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-tmp-dir\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.052898 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.052762 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-home\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.052898 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.052849 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-model-cache\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.054791 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.054768 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-dshm\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.055035 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.055015 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe0e284-ca30-495d-8462-c29e540c6bb7-tls-certs\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.074234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.074202 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mzp\" (UniqueName: \"kubernetes.io/projected/5fe0e284-ca30-495d-8462-c29e540c6bb7-kube-api-access-h4mzp\") pod \"test-model-c942c9-kserve-677956dbb7-p9cbj\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.196536 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.191293 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:15.529775 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.529743 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj"] Oct 31 01:01:15.531195 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:01:15.531165 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fe0e284_ca30_495d_8462_c29e540c6bb7.slice/crio-2435061ed6779d1ff407890ea439906f9c9106d348dcbb8cb19e07e09843cf36 WatchSource:0}: Error finding container 2435061ed6779d1ff407890ea439906f9c9106d348dcbb8cb19e07e09843cf36: Status 404 returned error can't find the container with id 2435061ed6779d1ff407890ea439906f9c9106d348dcbb8cb19e07e09843cf36 Oct 31 01:01:15.550395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:15.550348 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" event={"ID":"5fe0e284-ca30-495d-8462-c29e540c6bb7","Type":"ContainerStarted","Data":"2435061ed6779d1ff407890ea439906f9c9106d348dcbb8cb19e07e09843cf36"} Oct 31 01:01:16.318486 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:16.318036 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:16.560826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:16.560788 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" event={"ID":"5fe0e284-ca30-495d-8462-c29e540c6bb7","Type":"ContainerStarted","Data":"8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25"} Oct 31 01:01:16.600156 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:16.600053 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" podStartSLOduration=2.600038987 podStartE2EDuration="2.600038987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:16.59895209 +0000 UTC m=+2494.297065854" watchObservedRunningTime="2026-10-31 01:01:16.600038987 +0000 UTC m=+2494.298152753" Oct 31 01:01:17.192632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:17.192588 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:17.899010 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:17.898966 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:19.442894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:19.442850 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:20.460307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:20.460272 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:21.192550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:21.192512 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:22.914081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:22.914043 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:24.991511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:24.991414 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:25.192586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:25.192549 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:25.206868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:25.206838 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:25.650003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:25.649972 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:01:26.526231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:26.526191 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:28.211350 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:28.211313 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:29.812913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:29.812874 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:31.369624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:31.369586 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h"] Oct 31 01:01:31.370089 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:31.369797 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" containerID="cri-o://85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9" gracePeriod=30 Oct 31 01:01:31.414374 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:31.414337 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:33.090594 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:33.090541 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:33.187216 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:01:33.187170 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:01:34.618083 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:34.618045 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:36.197345 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.197311 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-97z9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:01:36.197771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.197377 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:01:36.354907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.354854 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:36.659855 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.659827 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 01:01:36.733848 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.733812 5978 generic.go:356] "Generic (PLEG): container finished" podID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerID="85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9" exitCode=0 Oct 31 01:01:36.734056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.733859 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" Oct 31 01:01:36.734056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.733902 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" event={"ID":"60da7d24-5e23-416c-9511-c8b9a7869ee9","Type":"ContainerDied","Data":"85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9"} Oct 31 01:01:36.734056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.733973 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h" event={"ID":"60da7d24-5e23-416c-9511-c8b9a7869ee9","Type":"ContainerDied","Data":"2cecc69b5dfed1e19e0cf0c70b0bb19d49d8ec366581f849f4ec1d604fba4166"} Oct 31 01:01:36.734056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.734003 5978 scope.go:122] "RemoveContainer" containerID="85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9" Oct 31 01:01:36.754613 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.754590 5978 scope.go:122] "RemoveContainer" containerID="85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9" Oct 31 01:01:36.755010 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:01:36.754987 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9\": container with ID starting with 85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9 not found: ID does not exist" containerID="85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9" Oct 31 01:01:36.755100 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.755019 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9"} err="failed to get container status \"85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9\": rpc error: code = NotFound desc = could not find container \"85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9\": container with ID starting with 85362079fab43a888d5895a93fbc9814330947090304128332f88b92da5356a9 not found: ID does not exist" Oct 31 01:01:36.800564 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.800534 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-token\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-token\") pod \"60da7d24-5e23-416c-9511-c8b9a7869ee9\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " Oct 31 01:01:36.800745 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.800612 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-podinfo\") pod \"60da7d24-5e23-416c-9511-c8b9a7869ee9\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " Oct 31 01:01:36.800784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.800753 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-envoy\") pod \"60da7d24-5e23-416c-9511-c8b9a7869ee9\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " Oct 31 01:01:36.800784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.800779 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-data\") pod \"60da7d24-5e23-416c-9511-c8b9a7869ee9\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " Oct 31 01:01:36.800849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.800812 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-credential-socket\") pod \"60da7d24-5e23-416c-9511-c8b9a7869ee9\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " Oct 31 01:01:36.800936 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.800924 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-kube-api-access-hgfnw\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-kube-api-access-hgfnw\") pod \"60da7d24-5e23-416c-9511-c8b9a7869ee9\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " Oct 31 01:01:36.800986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.800948 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-certs\") pod \"60da7d24-5e23-416c-9511-c8b9a7869ee9\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " Oct 31 01:01:36.801098 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.801085 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60da7d24-5e23-416c-9511-c8b9a7869ee9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60da7d24-5e23-416c-9511-c8b9a7869ee9-istiod-ca-cert\") pod \"60da7d24-5e23-416c-9511-c8b9a7869ee9\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " Oct 31 01:01:36.801174 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.801162 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-socket\") pod \"60da7d24-5e23-416c-9511-c8b9a7869ee9\" (UID: \"60da7d24-5e23-416c-9511-c8b9a7869ee9\") " Oct 31 01:01:36.859620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.859571 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-socket" pod "60da7d24-5e23-416c-9511-c8b9a7869ee9" (UID: "60da7d24-5e23-416c-9511-c8b9a7869ee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:36.860111 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.860061 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-data" pod "60da7d24-5e23-416c-9511-c8b9a7869ee9" (UID: "60da7d24-5e23-416c-9511-c8b9a7869ee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:36.860693 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.860669 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-certs" pod "60da7d24-5e23-416c-9511-c8b9a7869ee9" (UID: "60da7d24-5e23-416c-9511-c8b9a7869ee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:36.861674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.861654 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-token" pod "60da7d24-5e23-416c-9511-c8b9a7869ee9" (UID: "60da7d24-5e23-416c-9511-c8b9a7869ee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:36.862508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.862479 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-podinfo" pod "60da7d24-5e23-416c-9511-c8b9a7869ee9" (UID: "60da7d24-5e23-416c-9511-c8b9a7869ee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:01:36.862903 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.862885 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-credential-socket" pod "60da7d24-5e23-416c-9511-c8b9a7869ee9" (UID: "60da7d24-5e23-416c-9511-c8b9a7869ee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:36.863357 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.863333 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-kube-api-access-hgfnw" pod "60da7d24-5e23-416c-9511-c8b9a7869ee9" (UID: "60da7d24-5e23-416c-9511-c8b9a7869ee9"). InnerVolumeSpecName "kube-api-access-hgfnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:36.863797 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.863780 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-envoy" pod "60da7d24-5e23-416c-9511-c8b9a7869ee9" (UID: "60da7d24-5e23-416c-9511-c8b9a7869ee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:36.903417 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.903381 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:36.903417 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.903414 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:36.903647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.903428 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:36.903647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.903441 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:36.903647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.903465 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgfnw\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-kube-api-access-hgfnw\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:36.903647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.903475 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:36.903647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.903484 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60da7d24-5e23-416c-9511-c8b9a7869ee9-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:36.903647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.903492 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60da7d24-5e23-416c-9511-c8b9a7869ee9-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:36.952303 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:36.952251 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60da7d24-5e23-416c-9511-c8b9a7869ee9-istiod-ca-cert" pod "60da7d24-5e23-416c-9511-c8b9a7869ee9" (UID: "60da7d24-5e23-416c-9511-c8b9a7869ee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:37.005047 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:37.004969 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60da7d24-5e23-416c-9511-c8b9a7869ee9-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:37.081197 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:37.081161 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h"] Oct 31 01:01:37.085154 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:37.085123 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-97z9h"] Oct 31 01:01:37.901818 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:37.901777 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:38.430618 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:38.430580 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" path="/var/lib/kubelet/pods/60da7d24-5e23-416c-9511-c8b9a7869ee9/volumes" Oct 31 01:01:39.611034 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:39.610967 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:41.081851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:41.081817 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:41.511794 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:41.511620 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:41.668331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:41.668042 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:41.846161 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:41.846125 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:41.911475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:41.901602 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k"] Oct 31 01:01:41.936851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:41.936821 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6d97d-qtn5k"] Oct 31 01:01:42.361689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:42.361638 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:42.429135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:42.429100 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8b781a-5863-4aae-a186-e5a03aea7914" path="/var/lib/kubelet/pods/2f8b781a-5863-4aae-a186-e5a03aea7914/volumes" Oct 31 01:01:42.491429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:42.491391 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk"] Oct 31 01:01:42.500198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:42.500167 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd1dfc-7rzxk"] Oct 31 01:01:42.534284 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:42.534250 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:43.042019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:43.041980 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:44.223082 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:44.223044 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:44.427319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:44.427275 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a1f646-907c-4d5c-adbb-32dd752def1e" path="/var/lib/kubelet/pods/99a1f646-907c-4d5c-adbb-32dd752def1e/volumes" Oct 31 01:01:45.803291 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:45.803242 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:47.495097 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:47.495058 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:47.712902 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:47.712856 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:48.063149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:48.063117 5978 scope.go:122] "RemoveContainer" containerID="dca06491f6bf6784214dc287c9604be2c763f442dadc70fe2d5e3b4c9259e2f5" Oct 31 01:01:49.305752 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:49.305704 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:50.444210 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:50.444167 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:50.630485 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:50.630437 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:50.954067 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:50.954031 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:51.292648 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.292552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:51.520473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.517243 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:51.900873 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.900830 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns"] Oct 31 01:01:51.901826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.901795 5978 cpu_manager.go:414] "containerMap: removing container" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" Oct 31 01:01:51.901826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.901823 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" Oct 31 01:01:51.902046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.901958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:51.902046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.902033 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="60da7d24-5e23-416c-9511-c8b9a7869ee9" containerName="istio-proxy" Oct 31 01:01:51.907532 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.907508 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:51.923696 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.923664 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns"] Oct 31 01:01:51.928707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.928674 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:01:51.936763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:51.936734 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:01:52.005642 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.005610 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-609843-pjcns\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.005850 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.005661 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnbn\" (UniqueName: \"kubernetes.io/projected/4ffcc81b-b989-4db6-a372-0d77e5e35013-kube-api-access-xcnbn\") pod \"maas-api-revoke-keys-e2e-inf-b-609843-pjcns\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.005850 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.005717 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-609843-pjcns\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.107579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.107542 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-609843-pjcns\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.107783 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.107596 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnbn\" (UniqueName: \"kubernetes.io/projected/4ffcc81b-b989-4db6-a372-0d77e5e35013-kube-api-access-xcnbn\") pod \"maas-api-revoke-keys-e2e-inf-b-609843-pjcns\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.107783 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.107617 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-609843-pjcns\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.108346 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.108326 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-609843-pjcns\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.110090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.110072 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-609843-pjcns\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.132079 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.132039 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnbn\" (UniqueName: \"kubernetes.io/projected/4ffcc81b-b989-4db6-a372-0d77e5e35013-kube-api-access-xcnbn\") pod \"maas-api-revoke-keys-e2e-inf-b-609843-pjcns\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.232076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.231990 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:52.468397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.468290 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:52.644521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.639486 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns"] Oct 31 01:01:52.857026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.856992 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:52.922963 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:52.922927 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" event={"ID":"4ffcc81b-b989-4db6-a372-0d77e5e35013","Type":"ContainerStarted","Data":"52d3e689f70f42916676b4f7095d5eb18ca185783736132244b8342a4cbdc8dc"} Oct 31 01:01:53.755799 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:53.755752 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:53.933444 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:53.933402 5978 generic.go:356] "Generic (PLEG): container finished" podID="4ffcc81b-b989-4db6-a372-0d77e5e35013" containerID="e58b4f2227009657106706dffeb291b3b5e5a4ca66f22ae8b10f41193afdaf6a" exitCode=0 Oct 31 01:01:53.933699 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:53.933496 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" event={"ID":"4ffcc81b-b989-4db6-a372-0d77e5e35013","Type":"ContainerDied","Data":"e58b4f2227009657106706dffeb291b3b5e5a4ca66f22ae8b10f41193afdaf6a"} Oct 31 01:01:55.118520 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.118487 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:55.276811 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.276713 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ffcc81b-b989-4db6-a372-0d77e5e35013-kube-api-access-xcnbn\" (UniqueName: \"kubernetes.io/projected/4ffcc81b-b989-4db6-a372-0d77e5e35013-kube-api-access-xcnbn\") pod \"4ffcc81b-b989-4db6-a372-0d77e5e35013\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " Oct 31 01:01:55.277000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.276869 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-xks-ca\") pod \"4ffcc81b-b989-4db6-a372-0d77e5e35013\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " Oct 31 01:01:55.277000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.276898 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-openshift-ca\") pod \"4ffcc81b-b989-4db6-a372-0d77e5e35013\" (UID: \"4ffcc81b-b989-4db6-a372-0d77e5e35013\") " Oct 31 01:01:55.277302 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.277273 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-openshift-ca" pod "4ffcc81b-b989-4db6-a372-0d77e5e35013" (UID: "4ffcc81b-b989-4db6-a372-0d77e5e35013"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:55.277662 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.277641 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:55.279114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.279093 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-xks-ca" pod "4ffcc81b-b989-4db6-a372-0d77e5e35013" (UID: "4ffcc81b-b989-4db6-a372-0d77e5e35013"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:55.279204 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.279152 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffcc81b-b989-4db6-a372-0d77e5e35013-kube-api-access-xcnbn" pod "4ffcc81b-b989-4db6-a372-0d77e5e35013" (UID: "4ffcc81b-b989-4db6-a372-0d77e5e35013"). InnerVolumeSpecName "kube-api-access-xcnbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:55.379093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.379058 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffcc81b-b989-4db6-a372-0d77e5e35013-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:55.379093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.379091 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcnbn\" (UniqueName: \"kubernetes.io/projected/4ffcc81b-b989-4db6-a372-0d77e5e35013-kube-api-access-xcnbn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:01:55.686499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.686445 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:55.952674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.952582 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" Oct 31 01:01:55.952674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.952602 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns" event={"ID":"4ffcc81b-b989-4db6-a372-0d77e5e35013","Type":"ContainerDied","Data":"52d3e689f70f42916676b4f7095d5eb18ca185783736132244b8342a4cbdc8dc"} Oct 31 01:01:55.952674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:55.952655 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52d3e689f70f42916676b4f7095d5eb18ca185783736132244b8342a4cbdc8dc" Oct 31 01:01:56.782327 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:56.782284 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:57.065492 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:57.065389 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:57.914297 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:57.914258 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:58.732104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:58.732070 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:58.927746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:01:58.927706 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:00.104693 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:00.104656 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:01.955717 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:01.955677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:02.651861 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:02.651819 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:02.942925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:02.942834 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:03.043194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:03.043154 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf"] Oct 31 01:02:03.043750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:03.043412 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" containerID="cri-o://9dea4fbdac5f69de3a40ddf7ab813f2fc709796b35dcd7facdeba1170d2e88b7" gracePeriod=30 Oct 31 01:02:03.182975 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:02:03.182938 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:02:03.236538 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:03.224286 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:03.951684 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:03.951647 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:05.595955 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:05.595920 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:06.533805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:06.533767 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj"] Oct 31 01:02:06.534024 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:06.533986 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" containerID="cri-o://474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11" gracePeriod=30 Oct 31 01:02:07.197700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:07.197665 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:07.685575 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:07.685536 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:02:07.685781 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:07.685598 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:02:08.161665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.161629 5978 generic.go:356] "Generic (PLEG): container finished" podID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerID="9dea4fbdac5f69de3a40ddf7ab813f2fc709796b35dcd7facdeba1170d2e88b7" exitCode=0 Oct 31 01:02:08.161811 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.161719 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" event={"ID":"87e36a34-57af-4fa0-8adf-a5b67622a2a5","Type":"ContainerDied","Data":"9dea4fbdac5f69de3a40ddf7ab813f2fc709796b35dcd7facdeba1170d2e88b7"} Oct 31 01:02:08.230925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.230897 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 01:02:08.311599 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311516 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-socket\") pod \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " Oct 31 01:02:08.311599 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311560 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-data\") pod \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " Oct 31 01:02:08.311599 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311595 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-certs\") pod \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " Oct 31 01:02:08.311882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311722 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-token\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-token\") pod \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " Oct 31 01:02:08.311882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311787 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istiod-ca-cert\") pod \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " Oct 31 01:02:08.311972 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311888 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-kube-api-access-tz9z8\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-kube-api-access-tz9z8\") pod \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " Oct 31 01:02:08.311972 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311910 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-socket" pod "87e36a34-57af-4fa0-8adf-a5b67622a2a5" (UID: "87e36a34-57af-4fa0-8adf-a5b67622a2a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:08.311972 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311926 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-data" pod "87e36a34-57af-4fa0-8adf-a5b67622a2a5" (UID: "87e36a34-57af-4fa0-8adf-a5b67622a2a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:08.311972 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311935 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-envoy\") pod \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " Oct 31 01:02:08.311972 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.311940 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-certs" pod "87e36a34-57af-4fa0-8adf-a5b67622a2a5" (UID: "87e36a34-57af-4fa0-8adf-a5b67622a2a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:08.312223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.312008 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-credential-socket\") pod \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " Oct 31 01:02:08.312223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.312173 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-podinfo\") pod \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\" (UID: \"87e36a34-57af-4fa0-8adf-a5b67622a2a5\") " Oct 31 01:02:08.312325 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.312253 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istiod-ca-cert" pod "87e36a34-57af-4fa0-8adf-a5b67622a2a5" (UID: "87e36a34-57af-4fa0-8adf-a5b67622a2a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:08.312481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.312438 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-credential-socket" pod "87e36a34-57af-4fa0-8adf-a5b67622a2a5" (UID: "87e36a34-57af-4fa0-8adf-a5b67622a2a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:08.313112 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.313087 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:08.313319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.313302 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:08.313438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.313423 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:08.313571 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.313554 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:08.313701 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.313662 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:08.314722 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.314695 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-envoy" pod "87e36a34-57af-4fa0-8adf-a5b67622a2a5" (UID: "87e36a34-57af-4fa0-8adf-a5b67622a2a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:08.314941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.314915 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-kube-api-access-tz9z8" pod "87e36a34-57af-4fa0-8adf-a5b67622a2a5" (UID: "87e36a34-57af-4fa0-8adf-a5b67622a2a5"). InnerVolumeSpecName "kube-api-access-tz9z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:08.315060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.314967 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-podinfo" pod "87e36a34-57af-4fa0-8adf-a5b67622a2a5" (UID: "87e36a34-57af-4fa0-8adf-a5b67622a2a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:08.315158 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.315143 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-token" pod "87e36a34-57af-4fa0-8adf-a5b67622a2a5" (UID: "87e36a34-57af-4fa0-8adf-a5b67622a2a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:08.415663 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.415623 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz9z8\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-kube-api-access-tz9z8\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:08.415663 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.415661 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:08.415663 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.415673 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:08.415994 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.415686 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e36a34-57af-4fa0-8adf-a5b67622a2a5-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:08.881863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:08.881827 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:09.173394 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:09.173318 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" Oct 31 01:02:09.173633 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:09.173315 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf" event={"ID":"87e36a34-57af-4fa0-8adf-a5b67622a2a5","Type":"ContainerDied","Data":"1bf6d5f11c5eb9b3d98caa399bbd05784718207998a042f4d4c49679286328ad"} Oct 31 01:02:09.173633 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:09.173494 5978 scope.go:122] "RemoveContainer" containerID="9dea4fbdac5f69de3a40ddf7ab813f2fc709796b35dcd7facdeba1170d2e88b7" Oct 31 01:02:09.258427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:09.258390 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf"] Oct 31 01:02:09.267751 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:09.267719 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f2bdf3-openshift-default-cf4c47767-tv6bf"] Oct 31 01:02:10.426844 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:10.426806 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" path="/var/lib/kubelet/pods/87e36a34-57af-4fa0-8adf-a5b67622a2a5/volumes" Oct 31 01:02:10.481602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:10.481561 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:10.912506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:10.912465 5978 patch_prober.go:29] interesting pod/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:02:10.912661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:10.912531 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:02:11.827800 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.827771 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 01:02:11.973812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.973779 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-data\") pod \"9de4826a-0872-466a-a29e-1bd82254a45d\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " Oct 31 01:02:11.973992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.973830 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-certs\") pod \"9de4826a-0872-466a-a29e-1bd82254a45d\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " Oct 31 01:02:11.973992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.973916 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-envoy\") pod \"9de4826a-0872-466a-a29e-1bd82254a45d\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " Oct 31 01:02:11.973992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.973988 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-credential-socket\") pod \"9de4826a-0872-466a-a29e-1bd82254a45d\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " Oct 31 01:02:11.974268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974014 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-istio-token\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-istio-token\") pod \"9de4826a-0872-466a-a29e-1bd82254a45d\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " Oct 31 01:02:11.974268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974105 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-kube-api-access-dkfwq\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-kube-api-access-dkfwq\") pod \"9de4826a-0872-466a-a29e-1bd82254a45d\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " Oct 31 01:02:11.974268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974147 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9de4826a-0872-466a-a29e-1bd82254a45d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4826a-0872-466a-a29e-1bd82254a45d-istio-podinfo\") pod \"9de4826a-0872-466a-a29e-1bd82254a45d\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " Oct 31 01:02:11.974268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974149 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-data" pod "9de4826a-0872-466a-a29e-1bd82254a45d" (UID: "9de4826a-0872-466a-a29e-1bd82254a45d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.974268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974164 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-certs" pod "9de4826a-0872-466a-a29e-1bd82254a45d" (UID: "9de4826a-0872-466a-a29e-1bd82254a45d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.974268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974265 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de4826a-0872-466a-a29e-1bd82254a45d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4826a-0872-466a-a29e-1bd82254a45d-istiod-ca-cert\") pod \"9de4826a-0872-466a-a29e-1bd82254a45d\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " Oct 31 01:02:11.974634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974304 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-socket\") pod \"9de4826a-0872-466a-a29e-1bd82254a45d\" (UID: \"9de4826a-0872-466a-a29e-1bd82254a45d\") " Oct 31 01:02:11.974634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974425 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-credential-socket" pod "9de4826a-0872-466a-a29e-1bd82254a45d" (UID: "9de4826a-0872-466a-a29e-1bd82254a45d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.974803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974763 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-socket" pod "9de4826a-0872-466a-a29e-1bd82254a45d" (UID: "9de4826a-0872-466a-a29e-1bd82254a45d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.974871 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.974816 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de4826a-0872-466a-a29e-1bd82254a45d-istiod-ca-cert" pod "9de4826a-0872-466a-a29e-1bd82254a45d" (UID: "9de4826a-0872-466a-a29e-1bd82254a45d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:11.975250 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.975231 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.975287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.975259 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4826a-0872-466a-a29e-1bd82254a45d-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.975287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.975276 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.975353 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.975290 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.975353 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.975305 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:11.976465 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.976426 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9de4826a-0872-466a-a29e-1bd82254a45d-istio-podinfo" pod "9de4826a-0872-466a-a29e-1bd82254a45d" (UID: "9de4826a-0872-466a-a29e-1bd82254a45d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:11.976572 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.976535 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-istio-token" pod "9de4826a-0872-466a-a29e-1bd82254a45d" (UID: "9de4826a-0872-466a-a29e-1bd82254a45d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:11.976761 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.976742 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-envoy" pod "9de4826a-0872-466a-a29e-1bd82254a45d" (UID: "9de4826a-0872-466a-a29e-1bd82254a45d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:11.976838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:11.976747 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-kube-api-access-dkfwq" pod "9de4826a-0872-466a-a29e-1bd82254a45d" (UID: "9de4826a-0872-466a-a29e-1bd82254a45d"). InnerVolumeSpecName "kube-api-access-dkfwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:12.076653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.076617 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4826a-0872-466a-a29e-1bd82254a45d-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:12.076653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.076644 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:12.076653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.076654 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkfwq\" (UniqueName: \"kubernetes.io/projected/9de4826a-0872-466a-a29e-1bd82254a45d-kube-api-access-dkfwq\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:12.076653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.076663 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4826a-0872-466a-a29e-1bd82254a45d-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:02:12.201304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.201270 5978 generic.go:356] "Generic (PLEG): container finished" podID="9de4826a-0872-466a-a29e-1bd82254a45d" containerID="474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11" exitCode=0 Oct 31 01:02:12.201511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.201321 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" Oct 31 01:02:12.201511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.201359 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" event={"ID":"9de4826a-0872-466a-a29e-1bd82254a45d","Type":"ContainerDied","Data":"474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11"} Oct 31 01:02:12.201511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.201417 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj" event={"ID":"9de4826a-0872-466a-a29e-1bd82254a45d","Type":"ContainerDied","Data":"b27c5611a34684268ed7511a008877e6420043763ff15aa8e543886efa731cba"} Oct 31 01:02:12.201511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.201442 5978 scope.go:122] "RemoveContainer" containerID="474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11" Oct 31 01:02:12.221749 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.221730 5978 scope.go:122] "RemoveContainer" containerID="474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11" Oct 31 01:02:12.222124 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:02:12.222096 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11\": container with ID starting with 474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11 not found: ID does not exist" containerID="474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11" Oct 31 01:02:12.222183 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.222140 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11"} err="failed to get container status \"474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11\": rpc error: code = NotFound desc = could not find container \"474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11\": container with ID starting with 474bbb62a0b86a6fd0b28210b22fab52baa9d0c590cfa24b0c8a6f66f717bc11 not found: ID does not exist" Oct 31 01:02:12.267194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.267123 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj"] Oct 31 01:02:12.268754 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.268715 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-daaf26-openshift-default-64f4dbcc55-4fsjj"] Oct 31 01:02:12.317935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.317873 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:12.430549 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:12.430474 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" path="/var/lib/kubelet/pods/9de4826a-0872-466a-a29e-1bd82254a45d/volumes" Oct 31 01:02:13.981385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:13.981351 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:15.618477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:15.616270 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:17.434333 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:17.434290 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:19.106161 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:19.103808 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:21.016471 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:21.014024 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:22.939694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:22.939644 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:24.715968 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:24.715933 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:26.292768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:26.292730 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:26.462309 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:26.462266 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:28.112471 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:28.111201 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:29.849863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:29.849821 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:31.494022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:31.493968 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:32.063226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:32.063193 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:32.207485 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:32.207423 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:33.183265 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:02:33.183225 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:02:33.230753 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:33.230708 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:34.844905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:34.844858 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:36.461508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:36.461418 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:37.827773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:37.827739 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:38.468729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:38.468685 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:38.620705 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:38.620661 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns"] Oct 31 01:02:38.624124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:38.624091 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:38.628083 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:38.628058 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-609843-pjcns"] Oct 31 01:02:38.788497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:38.788385 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:39.231611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:39.231569 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:40.428626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:40.428586 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ffcc81b-b989-4db6-a372-0d77e5e35013" path="/var/lib/kubelet/pods/4ffcc81b-b989-4db6-a372-0d77e5e35013/volumes" Oct 31 01:02:40.818130 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:40.818090 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:42.544111 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:42.543706 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:44.253974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:44.253938 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:44.957971 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:44.957881 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.630197 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:45.630138 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.791016 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:45.790973 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.983824 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:45.983725 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:46.163677 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:46.163637 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:46.933496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:46.933442 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:47.645673 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.645632 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt"] Oct 31 01:02:47.649823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.649770 5978 cpu_manager.go:414] "containerMap: removing container" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" Oct 31 01:02:47.650101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.650080 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" Oct 31 01:02:47.650222 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.650211 5978 cpu_manager.go:414] "containerMap: removing container" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" Oct 31 01:02:47.656760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.656734 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" Oct 31 01:02:47.656910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.656898 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4ffcc81b-b989-4db6-a372-0d77e5e35013" containerName="revoke-keys" Oct 31 01:02:47.657994 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.657969 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ffcc81b-b989-4db6-a372-0d77e5e35013" containerName="revoke-keys" Oct 31 01:02:47.658864 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.658841 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de4826a-0872-466a-a29e-1bd82254a45d" containerName="istio-proxy" Oct 31 01:02:47.659071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.659057 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e36a34-57af-4fa0-8adf-a5b67622a2a5" containerName="istio-proxy" Oct 31 01:02:47.659177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.659164 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ffcc81b-b989-4db6-a372-0d77e5e35013" containerName="revoke-keys" Oct 31 01:02:47.683193 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.682666 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt"] Oct 31 01:02:47.683193 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.682821 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.715528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.709878 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-99a5c2-openshift-default-dockercfg-6l7wr\"" Oct 31 01:02:47.732161 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.732118 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:47.736495 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.736443 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-token\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.736867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.736692 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6v7\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-kube-api-access-2q6v7\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.736867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.736734 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-envoy\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.736867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.736841 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.737099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.736888 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-certs\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.737099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.736967 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-credential-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.737099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.737021 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163792a7-efc3-4a2b-96cf-a21b004b7f20-istiod-ca-cert\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.737226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.737164 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-podinfo\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.737226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.737214 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-data\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.838344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838300 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6v7\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-kube-api-access-2q6v7\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.838344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838344 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-envoy\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.838700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838391 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.838700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838433 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-certs\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.838700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838516 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-credential-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.838700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838569 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163792a7-efc3-4a2b-96cf-a21b004b7f20-istiod-ca-cert\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.838700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838637 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-podinfo\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.838700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838676 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-data\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.839012 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838809 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-token\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.839012 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838924 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.839012 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.838957 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-certs\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.839269 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.839238 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-credential-socket\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.839489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.839431 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-data\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.839696 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.839673 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163792a7-efc3-4a2b-96cf-a21b004b7f20-istiod-ca-cert\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.841037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.841009 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-envoy\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.841324 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.841299 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-podinfo\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.860392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.860357 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6v7\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-kube-api-access-2q6v7\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:47.860925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:47.860903 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-token\") pod \"e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:48.020772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.020672 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:48.033077 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.033045 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:48.216371 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.216319 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:48.363393 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.363368 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt"] Oct 31 01:02:48.365030 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:02:48.365001 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163792a7_efc3_4a2b_96cf_a21b004b7f20.slice/crio-0db3340fbc500f98614b101a44cc9d8f4db255b34c99c0ca66558fd47dfa07d7 WatchSource:0}: Error finding container 0db3340fbc500f98614b101a44cc9d8f4db255b34c99c0ca66558fd47dfa07d7: Status 404 returned error can't find the container with id 0db3340fbc500f98614b101a44cc9d8f4db255b34c99c0ca66558fd47dfa07d7 Oct 31 01:02:48.368230 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.368154 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:02:48.368335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.368288 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:02:48.368408 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.368341 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:02:48.513958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.513919 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" event={"ID":"163792a7-efc3-4a2b-96cf-a21b004b7f20","Type":"ContainerStarted","Data":"f5a5ba0a3503f2068ce94b0b9ca758f3768a15652f60fb2e7c0829965c3fe17a"} Oct 31 01:02:48.514232 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.513981 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" event={"ID":"163792a7-efc3-4a2b-96cf-a21b004b7f20","Type":"ContainerStarted","Data":"0db3340fbc500f98614b101a44cc9d8f4db255b34c99c0ca66558fd47dfa07d7"} Oct 31 01:02:48.559686 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:48.559618 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" podStartSLOduration=1.559597148 podStartE2EDuration="1.559597148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:48.557990335 +0000 UTC m=+2586.256104101" watchObservedRunningTime="2026-10-31 01:02:48.559597148 +0000 UTC m=+2586.257710915" Oct 31 01:02:49.033638 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:49.033598 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:49.040104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:49.040074 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:49.522650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:49.522612 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:49.523700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:49.523680 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:02:49.952658 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:49.952604 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:51.104681 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:51.104642 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:51.273993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:51.273962 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:51.779717 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:51.779675 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:52.800390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:52.800352 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:52.950680 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:52.950637 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:53.737542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:53.730419 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:55.660686 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:55.656626 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:57.408473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:57.408247 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:58.702169 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:58.702129 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:59.419170 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:02:59.419130 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:01.047767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:01.047727 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:02.135320 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:02.135282 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:02.606106 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:02.606065 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:03.183700 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:03:03.183664 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:03:04.196567 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:04.192005 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:05.846746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:05.846680 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:07.409698 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:07.409663 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:09.080500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:09.072240 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:10.742699 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:10.742603 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:12.289785 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:12.289744 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:13.992474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:13.985818 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:15.862767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:15.859115 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:17.133743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:17.133699 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:17.322807 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:17.322772 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:17.626610 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:17.626517 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:19.642979 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:19.642946 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:21.241305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:21.241271 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:22.979010 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:22.978950 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:24.805236 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:24.804900 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:26.550067 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:26.550016 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:28.283603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:28.283565 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:29.245834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:29.245797 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:29.705647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:29.705611 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:30.029331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:30.029232 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:30.305155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:30.305060 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:30.495994 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:30.495952 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:32.055231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:32.055189 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:32.228232 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:32.228194 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:33.183370 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:03:33.183327 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:03:33.928274 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:33.928233 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:34.122464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.122423 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:34.161217 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.161175 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b"] Oct 31 01:03:34.166685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.166658 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.190545 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.190441 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-99a5c2-serving-cert\"" Oct 31 01:03:34.203712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.203670 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b"] Oct 31 01:03:34.302483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.302432 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:34.345388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.345351 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.345617 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.345536 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9da597-737d-470f-bfd0-507c7940f1f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.345617 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.345592 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.345885 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.345859 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rgrr\" (UniqueName: \"kubernetes.io/projected/da9da597-737d-470f-bfd0-507c7940f1f4-kube-api-access-4rgrr\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.447598 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.447444 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9da597-737d-470f-bfd0-507c7940f1f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.447598 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.447531 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.447860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.447711 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rgrr\" (UniqueName: \"kubernetes.io/projected/da9da597-737d-470f-bfd0-507c7940f1f4-kube-api-access-4rgrr\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.447860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.447821 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.448335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.448305 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9da597-737d-470f-bfd0-507c7940f1f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.450281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.450255 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.450496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.450368 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.472789 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.472755 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rgrr\" (UniqueName: \"kubernetes.io/projected/da9da597-737d-470f-bfd0-507c7940f1f4-kube-api-access-4rgrr\") pod \"maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.482876 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.482832 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:34.490054 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.490023 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:34.789928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.789899 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b"] Oct 31 01:03:34.790499 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:03:34.790439 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9da597_737d_470f_bfd0_507c7940f1f4.slice/crio-6930aa5a03c8497d4a0cecf302b189d710e892f035da7562cf64b184cb6702fa WatchSource:0}: Error finding container 6930aa5a03c8497d4a0cecf302b189d710e892f035da7562cf64b184cb6702fa: Status 404 returned error can't find the container with id 6930aa5a03c8497d4a0cecf302b189d710e892f035da7562cf64b184cb6702fa Oct 31 01:03:34.792038 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.792016 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:03:34.919866 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:34.919832 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" event={"ID":"da9da597-737d-470f-bfd0-507c7940f1f4","Type":"ContainerStarted","Data":"6930aa5a03c8497d4a0cecf302b189d710e892f035da7562cf64b184cb6702fa"} Oct 31 01:03:35.929396 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:35.929355 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" event={"ID":"da9da597-737d-470f-bfd0-507c7940f1f4","Type":"ContainerStarted","Data":"d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3"} Oct 31 01:03:35.929849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:35.929489 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:35.981882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:35.981829 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" podStartSLOduration=1.62447102 podStartE2EDuration="1.981813317s" totalImagesPullingTime="357.342297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:35.9778102 +0000 UTC m=+2633.675923964" watchObservedRunningTime="2026-10-31 01:03:35.981813317 +0000 UTC m=+2633.679927081" Oct 31 01:03:36.230779 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:36.230678 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:36.430292 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:36.430258 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:38.216818 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:38.214681 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:39.914921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:39.914883 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:41.502343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:41.502304 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:41.944314 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:41.944281 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:03:43.104882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:43.104845 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:43.256638 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:43.256591 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m"] Oct 31 01:03:43.295036 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:43.294999 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs"] Oct 31 01:03:43.445900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:43.445820 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:43.611184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:43.611141 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:44.812351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:44.812313 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:46.613027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:46.612840 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:48.087857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.087818 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:48.257481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.257391 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" podUID="67a98371-3e68-422d-87d9-ee4afb582496" containerName="payload-processing" containerID="cri-o://858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f" gracePeriod=25 Oct 31 01:03:48.302770 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.296574 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" podUID="19fffbd8-b49b-4eaa-ae7c-42132790ca2b" containerName="payload-pre-processing" containerID="cri-o://cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4" gracePeriod=25 Oct 31 01:03:48.302770 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.301826 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:48.560205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.560174 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 01:03:48.644090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.644002 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a98371-3e68-422d-87d9-ee4afb582496-kube-api-access-j5zkx\" (UniqueName: \"kubernetes.io/projected/67a98371-3e68-422d-87d9-ee4afb582496-kube-api-access-j5zkx\") pod \"67a98371-3e68-422d-87d9-ee4afb582496\" (UID: \"67a98371-3e68-422d-87d9-ee4afb582496\") " Oct 31 01:03:48.644259 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.644177 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a98371-3e68-422d-87d9-ee4afb582496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a98371-3e68-422d-87d9-ee4afb582496-plugins-config-volume\") pod \"67a98371-3e68-422d-87d9-ee4afb582496\" (UID: \"67a98371-3e68-422d-87d9-ee4afb582496\") " Oct 31 01:03:48.644660 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.644632 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a98371-3e68-422d-87d9-ee4afb582496-plugins-config-volume" pod "67a98371-3e68-422d-87d9-ee4afb582496" (UID: "67a98371-3e68-422d-87d9-ee4afb582496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:48.645030 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.645000 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a98371-3e68-422d-87d9-ee4afb582496-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:03:48.646990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.646961 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a98371-3e68-422d-87d9-ee4afb582496-kube-api-access-j5zkx" pod "67a98371-3e68-422d-87d9-ee4afb582496" (UID: "67a98371-3e68-422d-87d9-ee4afb582496"). InnerVolumeSpecName "kube-api-access-j5zkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:48.746194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:48.746151 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5zkx\" (UniqueName: \"kubernetes.io/projected/67a98371-3e68-422d-87d9-ee4afb582496-kube-api-access-j5zkx\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:03:49.062692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:49.062437 5978 generic.go:356] "Generic (PLEG): container finished" podID="67a98371-3e68-422d-87d9-ee4afb582496" containerID="858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f" exitCode=0 Oct 31 01:03:49.062692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:49.062564 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" event={"ID":"67a98371-3e68-422d-87d9-ee4afb582496","Type":"ContainerDied","Data":"858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f"} Oct 31 01:03:49.062692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:49.062647 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" event={"ID":"67a98371-3e68-422d-87d9-ee4afb582496","Type":"ContainerDied","Data":"81f2930fe056d1462a99f3f3a92e2a04c2ac8bd1e77acd2946a0a6e61538e39f"} Oct 31 01:03:49.062692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:49.062578 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m" Oct 31 01:03:49.063036 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:49.062671 5978 scope.go:122] "RemoveContainer" containerID="858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f" Oct 31 01:03:49.074923 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:49.074902 5978 scope.go:122] "RemoveContainer" containerID="858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f" Oct 31 01:03:49.075243 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:03:49.075216 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f\": container with ID starting with 858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f not found: ID does not exist" containerID="858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f" Oct 31 01:03:49.075319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:49.075252 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f"} err="failed to get container status \"858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f\": rpc error: code = NotFound desc = could not find container \"858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f\": container with ID starting with 858553b02d257ee705f5068af1abb35260ac54f9f9ad8ed2830d186d1109464f not found: ID does not exist" Oct 31 01:03:49.110906 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:49.110868 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m"] Oct 31 01:03:49.119846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:49.119813 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-762753-64f6f4597c-vnz4m"] Oct 31 01:03:50.012430 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:50.012381 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:50.434565 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:50.434530 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a98371-3e68-422d-87d9-ee4afb582496" path="/var/lib/kubelet/pods/67a98371-3e68-422d-87d9-ee4afb582496/volumes" Oct 31 01:03:51.794813 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:51.794771 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:53.002664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:53.002625 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" podUID="19fffbd8-b49b-4eaa-ae7c-42132790ca2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:03:53.557408 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:53.557360 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:55.323991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:55.321048 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:56.882089 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:56.882045 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:58.147449 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.147413 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h"] Oct 31 01:03:58.148049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.147660 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" podUID="6a855609-b0fc-400a-a0bf-fe7295d4ac0b" containerName="maas-api" containerID="cri-o://2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc" gracePeriod=30 Oct 31 01:03:58.258893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.258857 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:58.501132 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.501095 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 01:03:58.619474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.619327 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-metrics-certs\") pod \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " Oct 31 01:03:58.619474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.619421 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-maas-api-tls\") pod \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " Oct 31 01:03:58.619764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.619573 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-kube-api-access-2hqwm\" (UniqueName: \"kubernetes.io/projected/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-kube-api-access-2hqwm\") pod \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " Oct 31 01:03:58.619764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.619678 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-openshift-service-ca\") pod \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\" (UID: \"6a855609-b0fc-400a-a0bf-fe7295d4ac0b\") " Oct 31 01:03:58.637912 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.637867 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-openshift-service-ca" pod "6a855609-b0fc-400a-a0bf-fe7295d4ac0b" (UID: "6a855609-b0fc-400a-a0bf-fe7295d4ac0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:58.662693 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.662645 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-metrics-certs" pod "6a855609-b0fc-400a-a0bf-fe7295d4ac0b" (UID: "6a855609-b0fc-400a-a0bf-fe7295d4ac0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:58.663496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.663437 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-kube-api-access-2hqwm" pod "6a855609-b0fc-400a-a0bf-fe7295d4ac0b" (UID: "6a855609-b0fc-400a-a0bf-fe7295d4ac0b"). InnerVolumeSpecName "kube-api-access-2hqwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:58.667989 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.667957 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-maas-api-tls" pod "6a855609-b0fc-400a-a0bf-fe7295d4ac0b" (UID: "6a855609-b0fc-400a-a0bf-fe7295d4ac0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:58.722341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.722304 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hqwm\" (UniqueName: \"kubernetes.io/projected/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-kube-api-access-2hqwm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:03:58.722341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.722339 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:03:58.722341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.722352 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:03:58.722731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.722360 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a855609-b0fc-400a-a0bf-fe7295d4ac0b-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:03:58.814726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.814626 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:58.980994 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:58.980956 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:59.148795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.148749 5978 generic.go:356] "Generic (PLEG): container finished" podID="6a855609-b0fc-400a-a0bf-fe7295d4ac0b" containerID="2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc" exitCode=0 Oct 31 01:03:59.149235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.148801 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" Oct 31 01:03:59.149235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.148815 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" event={"ID":"6a855609-b0fc-400a-a0bf-fe7295d4ac0b","Type":"ContainerDied","Data":"2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc"} Oct 31 01:03:59.149235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.148872 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h" event={"ID":"6a855609-b0fc-400a-a0bf-fe7295d4ac0b","Type":"ContainerDied","Data":"77a4e4014a60f5fb84ef53e433fe0abd68e564d83cf52843ac0a827543daebbd"} Oct 31 01:03:59.149235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.148896 5978 scope.go:122] "RemoveContainer" containerID="2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc" Oct 31 01:03:59.161131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.161111 5978 scope.go:122] "RemoveContainer" containerID="2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc" Oct 31 01:03:59.161599 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:03:59.161563 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc\": container with ID starting with 2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc not found: ID does not exist" containerID="2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc" Oct 31 01:03:59.161790 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.161606 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc"} err="failed to get container status \"2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc\": rpc error: code = NotFound desc = could not find container \"2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc\": container with ID starting with 2126923555c773f522f928bd1c0735c1b6f2a1fc27a89b70310094407ba5e6fc not found: ID does not exist" Oct 31 01:03:59.170773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.170742 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:59.201368 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.201332 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h"] Oct 31 01:03:59.210780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.210747 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-762753-78c86646fd-9wh5h"] Oct 31 01:03:59.488758 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:03:59.488666 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:00.427539 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:00.427500 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a855609-b0fc-400a-a0bf-fe7295d4ac0b" path="/var/lib/kubelet/pods/6a855609-b0fc-400a-a0bf-fe7295d4ac0b/volumes" Oct 31 01:04:00.896782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:00.896744 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:02.588535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:02.588500 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:03.002625 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:03.002528 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" podUID="19fffbd8-b49b-4eaa-ae7c-42132790ca2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:04:03.184112 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:04:03.184071 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:04:04.251211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:04.251112 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:05.558676 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:05.558634 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:06.077286 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:06.077212 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:06.528696 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:06.528595 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:06.767629 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:06.767583 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:06.896505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:06.888504 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm"] Oct 31 01:04:06.896505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:06.888790 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" containerID="cri-o://399f3add74301b379dc9d42a63f557e18257d95caee08f2df438074d45be52ee" gracePeriod=30 Oct 31 01:04:07.013474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.003694 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:07.533659 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.533619 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7"] Oct 31 01:04:07.534910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.534880 5978 cpu_manager.go:414] "containerMap: removing container" podUID="6a855609-b0fc-400a-a0bf-fe7295d4ac0b" containerName="maas-api" Oct 31 01:04:07.534910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.534909 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a855609-b0fc-400a-a0bf-fe7295d4ac0b" containerName="maas-api" Oct 31 01:04:07.535140 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.534980 5978 cpu_manager.go:414] "containerMap: removing container" podUID="67a98371-3e68-422d-87d9-ee4afb582496" containerName="payload-processing" Oct 31 01:04:07.535140 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.534989 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a98371-3e68-422d-87d9-ee4afb582496" containerName="payload-processing" Oct 31 01:04:07.535253 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.535192 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a98371-3e68-422d-87d9-ee4afb582496" containerName="payload-processing" Oct 31 01:04:07.535253 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.535220 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a855609-b0fc-400a-a0bf-fe7295d4ac0b" containerName="maas-api" Oct 31 01:04:07.542089 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.542024 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.575478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.569543 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:07.578639 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.578592 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:04:07.606427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.606391 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8t8\" (UniqueName: \"kubernetes.io/projected/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-kube-api-access-tj8t8\") pod \"maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.606926 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.606898 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.607602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.607577 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.614899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.614084 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7"] Oct 31 01:04:07.722530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.722086 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8t8\" (UniqueName: \"kubernetes.io/projected/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-kube-api-access-tj8t8\") pod \"maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.722530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.722160 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.722530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.722192 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.723328 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.723296 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.733491 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.732545 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.753188 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.753151 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8t8\" (UniqueName: \"kubernetes.io/projected/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-kube-api-access-tj8t8\") pod \"maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:07.879052 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:07.879016 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:08.204819 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.204784 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7"] Oct 31 01:04:08.204993 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:04:08.204960 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2371a7_beae_4547_bf7b_a52f7eb3b51f.slice/crio-84a45fda2ad97caa54176b06b2d30365cb93b85f1cba92e6e00aa8c50784ad6d WatchSource:0}: Error finding container 84a45fda2ad97caa54176b06b2d30365cb93b85f1cba92e6e00aa8c50784ad6d: Status 404 returned error can't find the container with id 84a45fda2ad97caa54176b06b2d30365cb93b85f1cba92e6e00aa8c50784ad6d Oct 31 01:04:08.275523 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.269740 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" event={"ID":"4b2371a7-beae-4547-bf7b-a52f7eb3b51f","Type":"ContainerStarted","Data":"84a45fda2ad97caa54176b06b2d30365cb93b85f1cba92e6e00aa8c50784ad6d"} Oct 31 01:04:08.296231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.296194 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:08.648324 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.648287 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 01:04:08.768799 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.768696 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-kube-api-access-txrxp\" (UniqueName: \"kubernetes.io/projected/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-kube-api-access-txrxp\") pod \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\" (UID: \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\") " Oct 31 01:04:08.768989 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.768961 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-plugins-config-volume\") pod \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\" (UID: \"19fffbd8-b49b-4eaa-ae7c-42132790ca2b\") " Oct 31 01:04:08.769447 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.769418 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-plugins-config-volume" pod "19fffbd8-b49b-4eaa-ae7c-42132790ca2b" (UID: "19fffbd8-b49b-4eaa-ae7c-42132790ca2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:08.769733 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.769704 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:08.771390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.771362 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-kube-api-access-txrxp" pod "19fffbd8-b49b-4eaa-ae7c-42132790ca2b" (UID: "19fffbd8-b49b-4eaa-ae7c-42132790ca2b"). InnerVolumeSpecName "kube-api-access-txrxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:08.870854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:08.870820 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txrxp\" (UniqueName: \"kubernetes.io/projected/19fffbd8-b49b-4eaa-ae7c-42132790ca2b-kube-api-access-txrxp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:09.281570 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.281530 5978 generic.go:356] "Generic (PLEG): container finished" podID="19fffbd8-b49b-4eaa-ae7c-42132790ca2b" containerID="cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4" exitCode=0 Oct 31 01:04:09.282020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.281580 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" Oct 31 01:04:09.282020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.281605 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" event={"ID":"19fffbd8-b49b-4eaa-ae7c-42132790ca2b","Type":"ContainerDied","Data":"cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4"} Oct 31 01:04:09.282020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.281665 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs" event={"ID":"19fffbd8-b49b-4eaa-ae7c-42132790ca2b","Type":"ContainerDied","Data":"919a721cd4c7c01984bb404ad005a0466777a7bdf9e99dc9df4a9dfd94b69fab"} Oct 31 01:04:09.282020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.281687 5978 scope.go:122] "RemoveContainer" containerID="cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4" Oct 31 01:04:09.284176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.284146 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" event={"ID":"4b2371a7-beae-4547-bf7b-a52f7eb3b51f","Type":"ContainerStarted","Data":"41487c71d750d75c6f1e458a7b6e97c76d5e37691d8cd755758917ecc0f304f0"} Oct 31 01:04:09.295341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.295321 5978 scope.go:122] "RemoveContainer" containerID="cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4" Oct 31 01:04:09.295717 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:04:09.295696 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4\": container with ID starting with cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4 not found: ID does not exist" containerID="cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4" Oct 31 01:04:09.295805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.295726 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4"} err="failed to get container status \"cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4\": rpc error: code = NotFound desc = could not find container \"cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4\": container with ID starting with cc37360e1a57b8dc899d55a355246cb4bd2bacb7fbb35cae4bbe9cc4264b16e4 not found: ID does not exist" Oct 31 01:04:09.365912 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.365847 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" podStartSLOduration=2.365834133 podStartE2EDuration="2.365834133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:09.364290087 +0000 UTC m=+2667.062403853" watchObservedRunningTime="2026-10-31 01:04:09.365834133 +0000 UTC m=+2667.063947898" Oct 31 01:04:09.400628 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.400594 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs"] Oct 31 01:04:09.407299 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:09.407269 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-762753-6847dc7c9f-xxdcs"] Oct 31 01:04:10.156879 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:10.156840 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:10.295524 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:10.295446 5978 generic.go:356] "Generic (PLEG): container finished" podID="4b2371a7-beae-4547-bf7b-a52f7eb3b51f" containerID="41487c71d750d75c6f1e458a7b6e97c76d5e37691d8cd755758917ecc0f304f0" exitCode=0 Oct 31 01:04:10.295920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:10.295537 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" event={"ID":"4b2371a7-beae-4547-bf7b-a52f7eb3b51f","Type":"ContainerDied","Data":"41487c71d750d75c6f1e458a7b6e97c76d5e37691d8cd755758917ecc0f304f0"} Oct 31 01:04:10.445602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:10.445521 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fffbd8-b49b-4eaa-ae7c-42132790ca2b" path="/var/lib/kubelet/pods/19fffbd8-b49b-4eaa-ae7c-42132790ca2b/volumes" Oct 31 01:04:11.336191 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.331690 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:11.557705 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.557630 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:11.742376 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.742327 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-kube-api-access-tj8t8\" (UniqueName: \"kubernetes.io/projected/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-kube-api-access-tj8t8\") pod \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " Oct 31 01:04:11.742611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.742498 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-xks-ca\") pod \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " Oct 31 01:04:11.742611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.742564 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-openshift-ca\") pod \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\" (UID: \"4b2371a7-beae-4547-bf7b-a52f7eb3b51f\") " Oct 31 01:04:11.743191 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.743166 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-openshift-ca" pod "4b2371a7-beae-4547-bf7b-a52f7eb3b51f" (UID: "4b2371a7-beae-4547-bf7b-a52f7eb3b51f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:11.743338 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.743319 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:11.744912 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.744877 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-xks-ca" pod "4b2371a7-beae-4547-bf7b-a52f7eb3b51f" (UID: "4b2371a7-beae-4547-bf7b-a52f7eb3b51f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:11.745072 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.744946 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-kube-api-access-tj8t8" pod "4b2371a7-beae-4547-bf7b-a52f7eb3b51f" (UID: "4b2371a7-beae-4547-bf7b-a52f7eb3b51f"). InnerVolumeSpecName "kube-api-access-tj8t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:11.790507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.788985 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:11.844920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.844874 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj8t8\" (UniqueName: \"kubernetes.io/projected/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-kube-api-access-tj8t8\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:11.845177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:11.845162 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2371a7-beae-4547-bf7b-a52f7eb3b51f-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.341963 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.341927 5978 generic.go:356] "Generic (PLEG): container finished" podID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerID="399f3add74301b379dc9d42a63f557e18257d95caee08f2df438074d45be52ee" exitCode=0 Oct 31 01:04:12.342512 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.342010 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" event={"ID":"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7","Type":"ContainerDied","Data":"399f3add74301b379dc9d42a63f557e18257d95caee08f2df438074d45be52ee"} Oct 31 01:04:12.342512 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.342133 5978 scope.go:122] "RemoveContainer" containerID="e45d73731f5054cc32759ec98f7b38cf8dba58020f970363d69cab75d5b132e6" Oct 31 01:04:12.344209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.344185 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" event={"ID":"4b2371a7-beae-4547-bf7b-a52f7eb3b51f","Type":"ContainerDied","Data":"84a45fda2ad97caa54176b06b2d30365cb93b85f1cba92e6e00aa8c50784ad6d"} Oct 31 01:04:12.344315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.344230 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84a45fda2ad97caa54176b06b2d30365cb93b85f1cba92e6e00aa8c50784ad6d" Oct 31 01:04:12.344315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.344237 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7" Oct 31 01:04:12.654635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.654593 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 01:04:12.756289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756258 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-kube-api-access-nz82l\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-kube-api-access-nz82l\") pod \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " Oct 31 01:04:12.756501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756308 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-credential-socket\") pod \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " Oct 31 01:04:12.756501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756383 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istiod-ca-cert\") pod \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " Oct 31 01:04:12.756501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756439 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-podinfo\") pod \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " Oct 31 01:04:12.756637 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756531 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-data\") pod \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " Oct 31 01:04:12.756711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756686 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-credential-socket" pod "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" (UID: "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.756841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756795 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-certs\") pod \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " Oct 31 01:04:12.756895 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756866 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-data" pod "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" (UID: "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.756895 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756879 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-token\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-token\") pod \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " Oct 31 01:04:12.757009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756902 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istiod-ca-cert" pod "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" (UID: "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:12.757009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.756937 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-socket\") pod \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " Oct 31 01:04:12.757131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.757018 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-envoy\") pod \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\" (UID: \"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7\") " Oct 31 01:04:12.757192 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.757126 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-certs" pod "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" (UID: "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.757234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.757186 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-socket" pod "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" (UID: "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.757869 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.757840 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.758008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.757871 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.758008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.757890 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.758008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.757904 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.758008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.757918 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.759439 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.759409 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-envoy" pod "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" (UID: "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:12.759767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.759689 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-kube-api-access-nz82l" pod "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" (UID: "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7"). InnerVolumeSpecName "kube-api-access-nz82l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:12.759767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.759734 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-podinfo" pod "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" (UID: "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:04:12.760129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.760109 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-token" pod "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" (UID: "8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:12.859172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.859064 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.859172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.859101 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.859172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.859114 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:12.859172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:12.859126 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz82l\" (UniqueName: \"kubernetes.io/projected/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7-kube-api-access-nz82l\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:13.353594 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:13.353563 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" Oct 31 01:04:13.354114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:13.353564 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm" event={"ID":"8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7","Type":"ContainerDied","Data":"44441d8136374ade5dfa0c5028a956cbc77b027f19b564085878824c3339b521"} Oct 31 01:04:13.354114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:13.353685 5978 scope.go:122] "RemoveContainer" containerID="399f3add74301b379dc9d42a63f557e18257d95caee08f2df438074d45be52ee" Oct 31 01:04:13.439670 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:13.439624 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm"] Oct 31 01:04:13.444948 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:13.444910 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-762753-openshift-default-85767dbb67-8dghm"] Oct 31 01:04:13.537291 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:13.529976 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:14.431523 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:14.431465 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" path="/var/lib/kubelet/pods/8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7/volumes" Oct 31 01:04:15.133165 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:15.133123 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:16.868028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:16.867968 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:17.920177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:17.920144 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:18.029062 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.029026 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj"] Oct 31 01:04:18.226571 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.226484 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:18.411518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.411483 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:18.421930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.421889 5978 generic.go:356] "Generic (PLEG): container finished" podID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerID="f5a5ba0a3503f2068ce94b0b9ca758f3768a15652f60fb2e7c0829965c3fe17a" exitCode=0 Oct 31 01:04:18.428224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.428187 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" event={"ID":"163792a7-efc3-4a2b-96cf-a21b004b7f20","Type":"ContainerDied","Data":"f5a5ba0a3503f2068ce94b0b9ca758f3768a15652f60fb2e7c0829965c3fe17a"} Oct 31 01:04:18.428752 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.428733 5978 scope.go:122] "RemoveContainer" containerID="f5a5ba0a3503f2068ce94b0b9ca758f3768a15652f60fb2e7c0829965c3fe17a" Oct 31 01:04:18.430926 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.430889 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:04:18.431098 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.430984 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:04:18.431098 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.431035 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:04:18.561408 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.561370 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld"] Oct 31 01:04:18.562787 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.562759 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" Oct 31 01:04:18.562787 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.562786 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" Oct 31 01:04:18.562980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.562868 5978 cpu_manager.go:414] "containerMap: removing container" podUID="19fffbd8-b49b-4eaa-ae7c-42132790ca2b" containerName="payload-pre-processing" Oct 31 01:04:18.562980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.562880 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fffbd8-b49b-4eaa-ae7c-42132790ca2b" containerName="payload-pre-processing" Oct 31 01:04:18.562980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.562932 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4b2371a7-beae-4547-bf7b-a52f7eb3b51f" containerName="revoke-keys" Oct 31 01:04:18.562980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.562940 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2371a7-beae-4547-bf7b-a52f7eb3b51f" containerName="revoke-keys" Oct 31 01:04:18.563210 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.563193 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" Oct 31 01:04:18.563251 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.563219 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" Oct 31 01:04:18.563251 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.563230 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fffbd8-b49b-4eaa-ae7c-42132790ca2b" containerName="payload-pre-processing" Oct 31 01:04:18.563251 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.563246 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2371a7-beae-4547-bf7b-a52f7eb3b51f" containerName="revoke-keys" Oct 31 01:04:18.566501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.566420 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.585020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.584991 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-b8fd5a-serving-cert\"" Oct 31 01:04:18.595001 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.594963 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld"] Oct 31 01:04:18.655038 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.655007 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e023e-3b05-4ca5-a4e2-19c682dcc570-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.655229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.655151 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.655229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.655221 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqc6x\" (UniqueName: \"kubernetes.io/projected/f01e023e-3b05-4ca5-a4e2-19c682dcc570-kube-api-access-kqc6x\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.655510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.655309 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.688241 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.688199 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:18.766787 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.766693 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e023e-3b05-4ca5-a4e2-19c682dcc570-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.767027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.766849 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.767027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.766924 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqc6x\" (UniqueName: \"kubernetes.io/projected/f01e023e-3b05-4ca5-a4e2-19c682dcc570-kube-api-access-kqc6x\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.767027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.767015 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.773213 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.771712 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e023e-3b05-4ca5-a4e2-19c682dcc570-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.809475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.793130 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.834490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.822761 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqc6x\" (UniqueName: \"kubernetes.io/projected/f01e023e-3b05-4ca5-a4e2-19c682dcc570-kube-api-access-kqc6x\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.834787 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.824108 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:18.859478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.847520 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:18.898521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:18.890745 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:19.024727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:19.024636 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:19.242602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:19.242558 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld"] Oct 31 01:04:19.444613 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:19.444575 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" event={"ID":"163792a7-efc3-4a2b-96cf-a21b004b7f20","Type":"ContainerStarted","Data":"3b536a3c52d10b7fb70a62ad0861231cd6ad0086a7076b46773cffe140818bc4"} Oct 31 01:04:19.446127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:19.446099 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" event={"ID":"f01e023e-3b05-4ca5-a4e2-19c682dcc570","Type":"ContainerStarted","Data":"1b3329c1b36f96c563c6773beefa8c77e690989ae45af13a7022b4a00db18617"} Oct 31 01:04:19.525620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:19.525576 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:04:20.039525 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:20.039143 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:20.040080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:20.040052 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:04:20.069696 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:20.059765 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:04:20.070017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:20.069981 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:04:20.455914 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:20.455875 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" event={"ID":"f01e023e-3b05-4ca5-a4e2-19c682dcc570","Type":"ContainerStarted","Data":"76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a"} Oct 31 01:04:20.456139 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:20.455987 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:20.497281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:20.497222 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" podStartSLOduration=2.088729398 podStartE2EDuration="2.497205438s" totalImagesPullingTime="408.47604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:20.493298103 +0000 UTC m=+2678.191411868" watchObservedRunningTime="2026-10-31 01:04:20.497205438 +0000 UTC m=+2678.195319204" Oct 31 01:04:20.852739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:20.852695 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:21.035478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:21.035079 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:04:21.035478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:21.035160 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:04:22.044501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:22.036860 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:04:22.044501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:22.036926 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:04:22.795207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:22.795174 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:23.032950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:23.032894 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" podUID="8732d41a-8b1b-49dc-b421-1df166cf5a74" containerName="payload-processing" containerID="cri-o://7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2" gracePeriod=25 Oct 31 01:04:23.042234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:23.042201 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:04:23.042901 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:23.042855 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:04:23.459434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:23.455827 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:23.901525 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:23.901488 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:24.036477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:24.036363 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:04:24.036477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:24.036427 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:04:24.596969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:24.596933 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:24.783004 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:24.782958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:25.144182 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:25.144145 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:25.244924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:25.244892 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:04:25.246576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:25.246545 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:04:26.038131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:26.038095 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx"] Oct 31 01:04:26.048330 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:26.048294 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f157460d-jh8hx"] Oct 31 01:04:26.210713 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:26.210672 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" podUID="8732d41a-8b1b-49dc-b421-1df166cf5a74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:04:26.428157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:26.428117 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2610e058-1053-48c7-a656-abe44618f8f6" path="/var/lib/kubelet/pods/2610e058-1053-48c7-a656-abe44618f8f6/volumes" Oct 31 01:04:26.471625 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:26.471597 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:04:27.032859 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:27.032818 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:28.644968 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:28.644931 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:28.799148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:28.799103 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:28.973316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:28.973219 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:30.344363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:30.344317 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:30.530580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:30.530541 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:30.752400 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:30.752299 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:31.174203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:31.174166 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:31.374781 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:31.374739 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:32.607137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:32.607095 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:33.194860 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:04:33.194822 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:04:34.311591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:34.311552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:35.947195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:35.947154 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:36.210684 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:36.210581 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" podUID="8732d41a-8b1b-49dc-b421-1df166cf5a74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:04:37.708222 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:37.708182 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:39.306073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:39.305978 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:40.985772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:40.985719 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:42.653180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:42.653138 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:43.463914 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.463882 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 01:04:43.513480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.510527 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8732d41a-8b1b-49dc-b421-1df166cf5a74-kube-api-access-skt2c\" (UniqueName: \"kubernetes.io/projected/8732d41a-8b1b-49dc-b421-1df166cf5a74-kube-api-access-skt2c\") pod \"8732d41a-8b1b-49dc-b421-1df166cf5a74\" (UID: \"8732d41a-8b1b-49dc-b421-1df166cf5a74\") " Oct 31 01:04:43.513480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.510600 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8732d41a-8b1b-49dc-b421-1df166cf5a74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8732d41a-8b1b-49dc-b421-1df166cf5a74-plugins-config-volume\") pod \"8732d41a-8b1b-49dc-b421-1df166cf5a74\" (UID: \"8732d41a-8b1b-49dc-b421-1df166cf5a74\") " Oct 31 01:04:43.513480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.512534 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8732d41a-8b1b-49dc-b421-1df166cf5a74-plugins-config-volume" pod "8732d41a-8b1b-49dc-b421-1df166cf5a74" (UID: "8732d41a-8b1b-49dc-b421-1df166cf5a74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:43.519011 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.518953 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8732d41a-8b1b-49dc-b421-1df166cf5a74-kube-api-access-skt2c" pod "8732d41a-8b1b-49dc-b421-1df166cf5a74" (UID: "8732d41a-8b1b-49dc-b421-1df166cf5a74"). InnerVolumeSpecName "kube-api-access-skt2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:43.612820 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.612728 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skt2c\" (UniqueName: \"kubernetes.io/projected/8732d41a-8b1b-49dc-b421-1df166cf5a74-kube-api-access-skt2c\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:43.612820 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.612756 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8732d41a-8b1b-49dc-b421-1df166cf5a74-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:04:43.878316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.878228 5978 generic.go:356] "Generic (PLEG): container finished" podID="8732d41a-8b1b-49dc-b421-1df166cf5a74" containerID="7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2" exitCode=0 Oct 31 01:04:43.878316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.878291 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" Oct 31 01:04:43.878853 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.878340 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" event={"ID":"8732d41a-8b1b-49dc-b421-1df166cf5a74","Type":"ContainerDied","Data":"7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2"} Oct 31 01:04:43.878853 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.878397 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj" event={"ID":"8732d41a-8b1b-49dc-b421-1df166cf5a74","Type":"ContainerDied","Data":"6ef5c68c25045005e318f8ad2e67d14b19f82c77727cfaa16d94bce85df5587e"} Oct 31 01:04:43.878853 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.878428 5978 scope.go:122] "RemoveContainer" containerID="7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2" Oct 31 01:04:43.894000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.893971 5978 scope.go:122] "RemoveContainer" containerID="7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2" Oct 31 01:04:43.894583 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:04:43.894498 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2\": container with ID starting with 7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2 not found: ID does not exist" containerID="7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2" Oct 31 01:04:43.894583 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.894542 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2"} err="failed to get container status \"7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2\": rpc error: code = NotFound desc = could not find container \"7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2\": container with ID starting with 7e09d036eeb403d5f92beb1bfdb53bf2f9998923fa5098fa3b7bcae4ecee42e2 not found: ID does not exist" Oct 31 01:04:43.928321 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.928286 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj"] Oct 31 01:04:43.931791 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:43.931769 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9de7f5-6b8cc55f75-txjxj"] Oct 31 01:04:44.114597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:44.114552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:44.487668 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:44.484683 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8732d41a-8b1b-49dc-b421-1df166cf5a74" path="/var/lib/kubelet/pods/8732d41a-8b1b-49dc-b421-1df166cf5a74/volumes" Oct 31 01:04:44.689801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:44.689767 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:44.841020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:44.840985 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:45.419136 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:45.419102 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:46.679873 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:46.679818 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:48.194964 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:48.194924 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:48.357128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:48.357094 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:49.028924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:49.028889 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5"] Oct 31 01:04:49.037723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:49.037698 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b022a6-k9hm5"] Oct 31 01:04:49.898552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:49.898510 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:50.426321 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:50.426281 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0675706-0707-4335-a6a7-3c73d90329cf" path="/var/lib/kubelet/pods/e0675706-0707-4335-a6a7-3c73d90329cf/volumes" Oct 31 01:04:50.493031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:50.492998 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:04:50.494001 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:50.493980 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:04:51.404630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:51.404585 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:51.873768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:51.873721 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:52.050034 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:52.049994 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:52.191407 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:52.191326 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7"] Oct 31 01:04:52.215882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:52.215844 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9de7f5-5tjw7"] Oct 31 01:04:52.274764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:52.274728 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:52.447337 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:52.447247 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:52.449791 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:52.449761 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2371a7-beae-4547-bf7b-a52f7eb3b51f" path="/var/lib/kubelet/pods/4b2371a7-beae-4547-bf7b-a52f7eb3b51f/volumes" Oct 31 01:04:52.765473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:52.765366 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:52.950803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:52.950765 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:54.325264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:54.325174 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:56.024295 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:56.023912 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:57.798384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:57.798347 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:59.495540 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:59.495493 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:59.915615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:04:59.915579 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.077367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:00.077335 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.325321 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:00.325282 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:01.336514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:01.328819 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:02.999353 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:02.999316 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:03.182067 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:05:03.182026 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:05:03.894813 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:03.894779 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:04.064020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:04.063981 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:05.020189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:05.020151 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:06.062263 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:06.062228 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:06.924239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:06.924193 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:07.510756 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:07.510718 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd"] Oct 31 01:05:07.511682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:07.511636 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" containerID="cri-o://f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72" gracePeriod=30 Oct 31 01:05:08.602609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:08.598671 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:10.219784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:10.219743 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:11.502884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:11.502842 5978 patch_prober.go:29] interesting pod/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:05:11.503296 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:11.502913 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:05:11.779771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:11.779682 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:11.920395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:11.920359 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:12.187276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.187236 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:12.891577 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.891549 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 01:05:12.968558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.968520 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-envoy\") pod \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " Oct 31 01:05:12.969137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.969111 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-socket\") pod \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " Oct 31 01:05:12.969441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.969412 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-kube-api-access-98cct\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-kube-api-access-98cct\") pod \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " Oct 31 01:05:12.969585 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.969507 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-credential-socket\") pod \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " Oct 31 01:05:12.969585 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.969576 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-data\") pod \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " Oct 31 01:05:12.969696 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.969615 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-token\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-token\") pod \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " Oct 31 01:05:12.969751 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.969712 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-certs\") pod \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " Oct 31 01:05:12.969803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.969778 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istiod-ca-cert\") pod \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " Oct 31 01:05:12.970112 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.969885 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-podinfo\") pod \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\" (UID: \"65fa924f-1429-48ed-8ca2-bb5a3fca1f92\") " Oct 31 01:05:12.976226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:12.976193 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-envoy" pod "65fa924f-1429-48ed-8ca2-bb5a3fca1f92" (UID: "65fa924f-1429-48ed-8ca2-bb5a3fca1f92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.014849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.013863 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-socket" pod "65fa924f-1429-48ed-8ca2-bb5a3fca1f92" (UID: "65fa924f-1429-48ed-8ca2-bb5a3fca1f92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.015040 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.014870 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-certs" pod "65fa924f-1429-48ed-8ca2-bb5a3fca1f92" (UID: "65fa924f-1429-48ed-8ca2-bb5a3fca1f92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.016225 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.016175 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-credential-socket" pod "65fa924f-1429-48ed-8ca2-bb5a3fca1f92" (UID: "65fa924f-1429-48ed-8ca2-bb5a3fca1f92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.025563 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.025521 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-podinfo" pod "65fa924f-1429-48ed-8ca2-bb5a3fca1f92" (UID: "65fa924f-1429-48ed-8ca2-bb5a3fca1f92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:05:13.025712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.025597 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istiod-ca-cert" pod "65fa924f-1429-48ed-8ca2-bb5a3fca1f92" (UID: "65fa924f-1429-48ed-8ca2-bb5a3fca1f92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:13.025712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.025599 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-kube-api-access-98cct" pod "65fa924f-1429-48ed-8ca2-bb5a3fca1f92" (UID: "65fa924f-1429-48ed-8ca2-bb5a3fca1f92"). InnerVolumeSpecName "kube-api-access-98cct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:13.025712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.025622 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-token" pod "65fa924f-1429-48ed-8ca2-bb5a3fca1f92" (UID: "65fa924f-1429-48ed-8ca2-bb5a3fca1f92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:13.031249 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.031213 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-data" pod "65fa924f-1429-48ed-8ca2-bb5a3fca1f92" (UID: "65fa924f-1429-48ed-8ca2-bb5a3fca1f92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:05:13.073126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.073089 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.073126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.073123 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.073126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.073133 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.073429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.073142 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98cct\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-kube-api-access-98cct\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.073429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.073154 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.073429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.073162 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.073429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.073172 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.073429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.073180 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.073429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.073187 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa924f-1429-48ed-8ca2-bb5a3fca1f92-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:13.177715 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.177596 5978 generic.go:356] "Generic (PLEG): container finished" podID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerID="f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72" exitCode=0 Oct 31 01:05:13.177921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.177777 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" event={"ID":"65fa924f-1429-48ed-8ca2-bb5a3fca1f92","Type":"ContainerDied","Data":"f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72"} Oct 31 01:05:13.177921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.177879 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" event={"ID":"65fa924f-1429-48ed-8ca2-bb5a3fca1f92","Type":"ContainerDied","Data":"366ceebf5dad1eda39ada126d9e084ec176ff3d7de43ba28f5e362a8ef9ad29f"} Oct 31 01:05:13.181601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.177924 5978 scope.go:122] "RemoveContainer" containerID="f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72" Oct 31 01:05:13.181601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.178160 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd" Oct 31 01:05:13.193645 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.193618 5978 scope.go:122] "RemoveContainer" containerID="f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d" Oct 31 01:05:13.205967 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.205945 5978 scope.go:122] "RemoveContainer" containerID="f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72" Oct 31 01:05:13.206264 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:05:13.206236 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72\": container with ID starting with f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72 not found: ID does not exist" containerID="f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72" Oct 31 01:05:13.206358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.206273 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72"} err="failed to get container status \"f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72\": rpc error: code = NotFound desc = could not find container \"f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72\": container with ID starting with f1d489cf03f2bb2c95c7c722829aeb00f037a6236a6d9c8b9f8f67f6108f2e72 not found: ID does not exist" Oct 31 01:05:13.206358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.206297 5978 scope.go:122] "RemoveContainer" containerID="f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d" Oct 31 01:05:13.206569 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:05:13.206553 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d\": container with ID starting with f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d not found: ID does not exist" containerID="f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d" Oct 31 01:05:13.206630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.206575 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d"} err="failed to get container status \"f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d\": rpc error: code = NotFound desc = could not find container \"f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d\": container with ID starting with f6054d382a95703ee46e9915bb59421eb5135c33b30d60035d44243eabe2979d not found: ID does not exist" Oct 31 01:05:13.233664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.233629 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd"] Oct 31 01:05:13.237191 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.237161 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-671959-openshift-default-586d4c9cc5-qxbwd"] Oct 31 01:05:13.560054 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:13.559941 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:14.427319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:14.427286 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" path="/var/lib/kubelet/pods/65fa924f-1429-48ed-8ca2-bb5a3fca1f92/volumes" Oct 31 01:05:14.533472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:14.532066 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:15.433575 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:15.433536 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:17.108568 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:17.106931 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:18.738284 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:18.733832 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:20.312861 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:20.312823 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:21.897822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:21.897788 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:23.515713 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:23.515671 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:23.986973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:23.986940 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:25.289503 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:25.289413 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:25.683897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:25.683860 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:25.857838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:25.857799 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:26.906607 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:26.906569 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:28.703312 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:28.703273 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:30.167798 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:30.167763 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:30.727571 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:30.727542 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:30.892585 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:30.892547 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:31.363823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:31.363785 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:33.184085 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:05:33.184045 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:05:33.940370 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:33.940336 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:34.470200 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:34.470111 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:36.123189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:36.123154 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:37.761563 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:37.761524 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4"] Oct 31 01:05:37.762387 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:37.762334 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" podUID="a1b4397e-f00d-402e-9cb2-161b2e8396bd" containerName="maas-api" containerID="cri-o://14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436" gracePeriod=30 Oct 31 01:05:37.879376 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:37.879339 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:38.387932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.387899 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 01:05:38.450264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.450223 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b4397e-f00d-402e-9cb2-161b2e8396bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b4397e-f00d-402e-9cb2-161b2e8396bd-openshift-service-ca\") pod \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " Oct 31 01:05:38.450438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.450306 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-maas-api-tls\") pod \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " Oct 31 01:05:38.450438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.450398 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-metrics-certs\") pod \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " Oct 31 01:05:38.450624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.450494 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b4397e-f00d-402e-9cb2-161b2e8396bd-kube-api-access-rpvbl\" (UniqueName: \"kubernetes.io/projected/a1b4397e-f00d-402e-9cb2-161b2e8396bd-kube-api-access-rpvbl\") pod \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\" (UID: \"a1b4397e-f00d-402e-9cb2-161b2e8396bd\") " Oct 31 01:05:38.451050 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.450714 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b4397e-f00d-402e-9cb2-161b2e8396bd-openshift-service-ca" pod "a1b4397e-f00d-402e-9cb2-161b2e8396bd" (UID: "a1b4397e-f00d-402e-9cb2-161b2e8396bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:38.462173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.462128 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-metrics-certs" pod "a1b4397e-f00d-402e-9cb2-161b2e8396bd" (UID: "a1b4397e-f00d-402e-9cb2-161b2e8396bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:38.462358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.462231 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-maas-api-tls" pod "a1b4397e-f00d-402e-9cb2-161b2e8396bd" (UID: "a1b4397e-f00d-402e-9cb2-161b2e8396bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:38.467391 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.467348 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b4397e-f00d-402e-9cb2-161b2e8396bd-kube-api-access-rpvbl" pod "a1b4397e-f00d-402e-9cb2-161b2e8396bd" (UID: "a1b4397e-f00d-402e-9cb2-161b2e8396bd"). InnerVolumeSpecName "kube-api-access-rpvbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:38.495124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.495084 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:38.497238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.497203 5978 generic.go:356] "Generic (PLEG): container finished" podID="a1b4397e-f00d-402e-9cb2-161b2e8396bd" containerID="14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436" exitCode=0 Oct 31 01:05:38.497389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.497352 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" event={"ID":"a1b4397e-f00d-402e-9cb2-161b2e8396bd","Type":"ContainerDied","Data":"14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436"} Oct 31 01:05:38.497478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.497407 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" event={"ID":"a1b4397e-f00d-402e-9cb2-161b2e8396bd","Type":"ContainerDied","Data":"01814ecdbe7c5f374945e6037c707535ff1b2ae751139e5e2edd64338838a0ae"} Oct 31 01:05:38.497547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.497443 5978 scope.go:122] "RemoveContainer" containerID="14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436" Oct 31 01:05:38.498481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.497745 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4" Oct 31 01:05:38.526148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.526122 5978 scope.go:122] "RemoveContainer" containerID="14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436" Oct 31 01:05:38.526566 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:05:38.526542 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436\": container with ID starting with 14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436 not found: ID does not exist" containerID="14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436" Oct 31 01:05:38.526665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.526579 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436"} err="failed to get container status \"14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436\": rpc error: code = NotFound desc = could not find container \"14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436\": container with ID starting with 14d78e8dfaaa996ebef4b1b2d41ce44f1b78cccf190215fb2799c02552bf0436 not found: ID does not exist" Oct 31 01:05:38.552984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.552825 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b4397e-f00d-402e-9cb2-161b2e8396bd-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:38.552984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.552866 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:38.552984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.552884 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b4397e-f00d-402e-9cb2-161b2e8396bd-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:38.552984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.552897 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpvbl\" (UniqueName: \"kubernetes.io/projected/a1b4397e-f00d-402e-9cb2-161b2e8396bd-kube-api-access-rpvbl\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:05:38.554762 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.554718 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4"] Oct 31 01:05:38.558627 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.558601 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72120c-5d47cf97f5-rhln4"] Oct 31 01:05:38.642744 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:38.642707 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:39.138863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:39.138827 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:39.286149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:39.286109 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:39.525357 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:39.525193 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:40.405195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:40.405164 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:40.427488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:40.427433 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b4397e-f00d-402e-9cb2-161b2e8396bd" path="/var/lib/kubelet/pods/a1b4397e-f00d-402e-9cb2-161b2e8396bd/volumes" Oct 31 01:05:42.092592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:42.092550 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:43.656564 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:43.656523 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:44.041005 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:44.040906 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj"] Oct 31 01:05:44.050137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:44.050103 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8706aaa1-lrwcj"] Oct 31 01:05:44.427568 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:44.427532 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc0281b-df0d-4669-b967-b75e964e1ce4" path="/var/lib/kubelet/pods/cdc0281b-df0d-4669-b967-b75e964e1ce4/volumes" Oct 31 01:05:45.427323 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:45.427277 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:45.885909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:45.885867 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:46.061661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:46.061624 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:46.250245 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:46.250145 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:47.289735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:47.289693 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:48.202692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:48.202655 5978 scope.go:122] "RemoveContainer" containerID="f3561bd54f2c2e06ce3d5c7b23fcaeb7a91d69c50e64239f2ca6216d85449352" Oct 31 01:05:48.217246 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:48.217216 5978 scope.go:122] "RemoveContainer" containerID="98c811f1b6a1a5e188fac637d5357cd81e4689e47424f9ff4de5f81e79ea0a5f" Oct 31 01:05:48.360119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:48.360070 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:49.244057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:49.244012 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:50.880123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:50.880084 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:52.517527 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:52.517488 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:54.265066 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:54.265033 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:55.183991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:55.183955 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:55.706071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:55.706012 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:56.130285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:56.130248 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:57.516249 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:57.516201 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:57.716586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:57.716548 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:57.891073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:57.891037 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:59.402999 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:05:59.402958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:00.995326 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:00.995282 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:02.588513 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:02.588443 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:03.184873 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:06:03.184834 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:06:04.272365 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:04.272306 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:05.812829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:05.811302 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:07.393345 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:07.393308 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:08.979482 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:08.979416 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:10.642817 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:10.642776 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:12.178423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:12.178381 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:13.883128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:13.883092 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:15.512380 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:15.512334 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:17.314160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:17.314120 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:18.920065 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:18.920027 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:20.573502 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:20.573467 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:22.203494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:22.203419 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:23.721200 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:23.721147 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:23.885981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:23.885888 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:25.533026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:25.532989 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:26.139052 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:26.139016 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:27.061891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:27.061854 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:27.594761 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:27.594725 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:27.755299 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:27.755261 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:28.236313 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:28.236273 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:29.401729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:29.401594 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:31.185346 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:31.185309 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:32.779314 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:32.779273 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:33.214467 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:06:33.214410 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:06:33.283365 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:33.283328 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:34.853166 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:34.853126 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:35.002123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:35.002086 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:36.136867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:36.136826 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:36.353435 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:36.353399 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:36.492862 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:36.492780 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt"] Oct 31 01:06:36.493086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:36.493060 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" containerID="cri-o://521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3" gracePeriod=30 Oct 31 01:06:36.589472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:36.589421 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:36.772748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:36.772644 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:37.393179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:37.393140 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:37.956825 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:37.956777 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:38.165331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:38.165289 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:38.616135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:38.616101 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:40.305396 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:40.305354 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:40.566192 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:40.566095 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:41.312909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.312860 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:41.483520 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.483485 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:41.775360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.775330 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 01:06:41.857101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.857009 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-token\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-token\") pod \"6da83d69-b959-4613-9b4d-d869ddb1884c\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " Oct 31 01:06:41.857101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.857049 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da83d69-b959-4613-9b4d-d869ddb1884c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da83d69-b959-4613-9b4d-d869ddb1884c-istiod-ca-cert\") pod \"6da83d69-b959-4613-9b4d-d869ddb1884c\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " Oct 31 01:06:41.857101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.857093 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-credential-socket\") pod \"6da83d69-b959-4613-9b4d-d869ddb1884c\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " Oct 31 01:06:41.857381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.857133 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-data\") pod \"6da83d69-b959-4613-9b4d-d869ddb1884c\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " Oct 31 01:06:41.857381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.857198 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-socket\") pod \"6da83d69-b959-4613-9b4d-d869ddb1884c\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " Oct 31 01:06:41.857381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.857272 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-envoy\") pod \"6da83d69-b959-4613-9b4d-d869ddb1884c\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " Oct 31 01:06:41.857543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.857397 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-kube-api-access-4pcrf\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-kube-api-access-4pcrf\") pod \"6da83d69-b959-4613-9b4d-d869ddb1884c\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " Oct 31 01:06:41.857543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.857464 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-podinfo\") pod \"6da83d69-b959-4613-9b4d-d869ddb1884c\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " Oct 31 01:06:41.857543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.857515 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-certs\") pod \"6da83d69-b959-4613-9b4d-d869ddb1884c\" (UID: \"6da83d69-b959-4613-9b4d-d869ddb1884c\") " Oct 31 01:06:41.858130 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.858082 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-data" pod "6da83d69-b959-4613-9b4d-d869ddb1884c" (UID: "6da83d69-b959-4613-9b4d-d869ddb1884c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:41.858293 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.858260 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da83d69-b959-4613-9b4d-d869ddb1884c-istiod-ca-cert" pod "6da83d69-b959-4613-9b4d-d869ddb1884c" (UID: "6da83d69-b959-4613-9b4d-d869ddb1884c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:41.858293 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.858179 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-socket" pod "6da83d69-b959-4613-9b4d-d869ddb1884c" (UID: "6da83d69-b959-4613-9b4d-d869ddb1884c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:41.858293 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.858206 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-credential-socket" pod "6da83d69-b959-4613-9b4d-d869ddb1884c" (UID: "6da83d69-b959-4613-9b4d-d869ddb1884c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:41.858616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.858262 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-certs" pod "6da83d69-b959-4613-9b4d-d869ddb1884c" (UID: "6da83d69-b959-4613-9b4d-d869ddb1884c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:41.860651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.860624 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-token" pod "6da83d69-b959-4613-9b4d-d869ddb1884c" (UID: "6da83d69-b959-4613-9b4d-d869ddb1884c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:41.861116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.861091 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-podinfo" pod "6da83d69-b959-4613-9b4d-d869ddb1884c" (UID: "6da83d69-b959-4613-9b4d-d869ddb1884c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:06:41.861221 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.861089 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-envoy" pod "6da83d69-b959-4613-9b4d-d869ddb1884c" (UID: "6da83d69-b959-4613-9b4d-d869ddb1884c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:41.861314 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.861299 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-kube-api-access-4pcrf" pod "6da83d69-b959-4613-9b4d-d869ddb1884c" (UID: "6da83d69-b959-4613-9b4d-d869ddb1884c"). InnerVolumeSpecName "kube-api-access-4pcrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:41.959786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.959746 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.959786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.959779 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.959786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.959793 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.960049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.959804 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da83d69-b959-4613-9b4d-d869ddb1884c-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.960049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.959816 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.960049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.959828 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.960049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.959840 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.960049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.959850 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da83d69-b959-4613-9b4d-d869ddb1884c-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:06:41.960049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:41.959861 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pcrf\" (UniqueName: \"kubernetes.io/projected/6da83d69-b959-4613-9b4d-d869ddb1884c-kube-api-access-4pcrf\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:06:42.089428 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.089396 5978 generic.go:356] "Generic (PLEG): container finished" podID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerID="521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3" exitCode=0 Oct 31 01:06:42.089628 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.089461 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" Oct 31 01:06:42.089628 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.089473 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" event={"ID":"6da83d69-b959-4613-9b4d-d869ddb1884c","Type":"ContainerDied","Data":"521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3"} Oct 31 01:06:42.089628 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.089516 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt" event={"ID":"6da83d69-b959-4613-9b4d-d869ddb1884c","Type":"ContainerDied","Data":"5881f062234050006258de6efb47368c8f6c1314471dfb8a9e16c3eac09a60a0"} Oct 31 01:06:42.089628 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.089538 5978 scope.go:122] "RemoveContainer" containerID="521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3" Oct 31 01:06:42.102133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.102111 5978 scope.go:122] "RemoveContainer" containerID="72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40" Oct 31 01:06:42.113317 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.113294 5978 scope.go:122] "RemoveContainer" containerID="521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3" Oct 31 01:06:42.113609 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:06:42.113589 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3\": container with ID starting with 521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3 not found: ID does not exist" containerID="521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3" Oct 31 01:06:42.113661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.113620 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3"} err="failed to get container status \"521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3\": rpc error: code = NotFound desc = could not find container \"521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3\": container with ID starting with 521b8f7e46822c2a60d69e8dd6ce9434bc01d945e8f28ac38b50002dd758ebb3 not found: ID does not exist" Oct 31 01:06:42.113661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.113638 5978 scope.go:122] "RemoveContainer" containerID="72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40" Oct 31 01:06:42.113946 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:06:42.113921 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40\": container with ID starting with 72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40 not found: ID does not exist" containerID="72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40" Oct 31 01:06:42.114019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.113953 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40"} err="failed to get container status \"72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40\": rpc error: code = NotFound desc = could not find container \"72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40\": container with ID starting with 72db436c538e333da98adce5a24922fcbc84b6881f8e3e282a72d56a2571ff40 not found: ID does not exist" Oct 31 01:06:42.137033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.136998 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt"] Oct 31 01:06:42.145944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.145915 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-82b5e8-openshift-default-79b8ff867c-gmlkt"] Oct 31 01:06:42.196945 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.196857 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:42.451841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:42.451757 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" path="/var/lib/kubelet/pods/6da83d69-b959-4613-9b4d-d869ddb1884c/volumes" Oct 31 01:06:43.024397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:43.024361 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:43.169099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:43.169059 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:43.885000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:43.884961 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:44.515056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:44.515024 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:45.696917 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:45.696874 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:47.497955 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:47.497918 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:48.276133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:48.276103 5978 scope.go:122] "RemoveContainer" containerID="43b9b6b0499d64e478ed1da4aa40829c592a9b95a64212d19faddb83edeef69f" Oct 31 01:06:48.286743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:48.286723 5978 scope.go:122] "RemoveContainer" containerID="41c61425dff61151f6aaa526b4c2e26f623ef56d26db2a80b8ad47e6a50b7ffc" Oct 31 01:06:48.480481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:48.480422 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:49.165074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:49.165039 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:50.830902 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:50.830855 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:52.398282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:52.398224 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:54.059989 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:54.059953 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:55.369884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:55.369847 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:55.507293 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:55.507256 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:55.704692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:55.704603 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:57.338367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:57.338321 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:59.010610 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:06:59.010573 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:00.601755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:00.601719 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:02.313372 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:02.313327 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:03.182642 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:07:03.182597 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:07:03.792695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:03.792658 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:05.349477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:05.347852 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:07.080436 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:07.080396 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:08.824762 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:08.824705 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:10.615096 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:10.615059 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:12.261007 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:12.260964 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:13.637402 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:13.637362 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:14.140946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:14.140915 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:14.301136 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:14.301069 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:14.776509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:14.776442 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:16.000692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:16.000650 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:16.160291 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:16.160248 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:17.717492 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:17.717340 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:19.499593 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:19.499555 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:20.831003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:20.830955 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj"] Oct 31 01:07:20.898515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:20.898445 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:21.416160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:21.416121 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:21.613713 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:21.613672 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:21.763675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:21.763574 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:21.948144 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:21.948104 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:23.208074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:23.208039 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:24.554624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:24.554588 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:24.840385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:24.840349 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:26.483757 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:26.483718 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:28.117095 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:28.117046 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:29.785192 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:29.785150 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:31.300951 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:31.300841 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:32.877488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:32.877439 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:33.156849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:33.156754 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:33.182925 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:07:33.182890 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:07:33.305955 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:33.305923 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:34.597099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:34.597053 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:35.885039 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:35.885001 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" podUID="5fe0e284-ca30-495d-8462-c29e540c6bb7" containerName="main" containerID="cri-o://8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25" gracePeriod=45 Oct 31 01:07:36.129263 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.129233 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:36.173550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.173522 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:07:36.320711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.320677 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-dshm\") pod \"5fe0e284-ca30-495d-8462-c29e540c6bb7\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " Oct 31 01:07:36.320711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.320718 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-model-cache\") pod \"5fe0e284-ca30-495d-8462-c29e540c6bb7\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " Oct 31 01:07:36.320993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.320833 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-home\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-home\") pod \"5fe0e284-ca30-495d-8462-c29e540c6bb7\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " Oct 31 01:07:36.320993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.320879 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe0e284-ca30-495d-8462-c29e540c6bb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe0e284-ca30-495d-8462-c29e540c6bb7-tls-certs\") pod \"5fe0e284-ca30-495d-8462-c29e540c6bb7\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " Oct 31 01:07:36.320993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.320956 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-tmp-dir\") pod \"5fe0e284-ca30-495d-8462-c29e540c6bb7\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " Oct 31 01:07:36.321168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.320998 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe0e284-ca30-495d-8462-c29e540c6bb7-kube-api-access-h4mzp\" (UniqueName: \"kubernetes.io/projected/5fe0e284-ca30-495d-8462-c29e540c6bb7-kube-api-access-h4mzp\") pod \"5fe0e284-ca30-495d-8462-c29e540c6bb7\" (UID: \"5fe0e284-ca30-495d-8462-c29e540c6bb7\") " Oct 31 01:07:36.321168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.321034 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-home" pod "5fe0e284-ca30-495d-8462-c29e540c6bb7" (UID: "5fe0e284-ca30-495d-8462-c29e540c6bb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:36.321168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.321050 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-model-cache" pod "5fe0e284-ca30-495d-8462-c29e540c6bb7" (UID: "5fe0e284-ca30-495d-8462-c29e540c6bb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:36.321340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.321224 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-tmp-dir" pod "5fe0e284-ca30-495d-8462-c29e540c6bb7" (UID: "5fe0e284-ca30-495d-8462-c29e540c6bb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:36.321574 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.321556 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:36.321675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.321576 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:36.321675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.321613 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:36.323144 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.323122 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe0e284-ca30-495d-8462-c29e540c6bb7-tls-certs" pod "5fe0e284-ca30-495d-8462-c29e540c6bb7" (UID: "5fe0e284-ca30-495d-8462-c29e540c6bb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:36.323243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.323171 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe0e284-ca30-495d-8462-c29e540c6bb7-kube-api-access-h4mzp" pod "5fe0e284-ca30-495d-8462-c29e540c6bb7" (UID: "5fe0e284-ca30-495d-8462-c29e540c6bb7"). InnerVolumeSpecName "kube-api-access-h4mzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:36.323243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.323201 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-dshm" pod "5fe0e284-ca30-495d-8462-c29e540c6bb7" (UID: "5fe0e284-ca30-495d-8462-c29e540c6bb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:36.423691 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.423603 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe0e284-ca30-495d-8462-c29e540c6bb7-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:36.423691 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.423636 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe0e284-ca30-495d-8462-c29e540c6bb7-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:36.423691 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.423652 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4mzp\" (UniqueName: \"kubernetes.io/projected/5fe0e284-ca30-495d-8462-c29e540c6bb7-kube-api-access-h4mzp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:36.524955 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.524918 5978 generic.go:356] "Generic (PLEG): container finished" podID="5fe0e284-ca30-495d-8462-c29e540c6bb7" containerID="8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25" exitCode=0 Oct 31 01:07:36.525129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.524964 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" Oct 31 01:07:36.525129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.524996 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" event={"ID":"5fe0e284-ca30-495d-8462-c29e540c6bb7","Type":"ContainerDied","Data":"8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25"} Oct 31 01:07:36.525129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.525041 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj" event={"ID":"5fe0e284-ca30-495d-8462-c29e540c6bb7","Type":"ContainerDied","Data":"2435061ed6779d1ff407890ea439906f9c9106d348dcbb8cb19e07e09843cf36"} Oct 31 01:07:36.525129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.525064 5978 scope.go:122] "RemoveContainer" containerID="8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25" Oct 31 01:07:36.535733 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.535708 5978 scope.go:122] "RemoveContainer" containerID="8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25" Oct 31 01:07:36.536012 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:07:36.535984 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25\": container with ID starting with 8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25 not found: ID does not exist" containerID="8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25" Oct 31 01:07:36.536119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.536016 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25"} err="failed to get container status \"8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25\": rpc error: code = NotFound desc = could not find container \"8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25\": container with ID starting with 8903741bc077557d3c84436940aeeff5991fbbfa7cdc17136f2844b1ee90af25 not found: ID does not exist" Oct 31 01:07:36.570426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.570393 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj"] Oct 31 01:07:36.575961 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:36.575935 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c942c9/test-model-c942c9-kserve-677956dbb7-p9cbj"] Oct 31 01:07:37.722321 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:37.722276 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:38.427598 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:38.427564 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe0e284-ca30-495d-8462-c29e540c6bb7" path="/var/lib/kubelet/pods/5fe0e284-ca30-495d-8462-c29e540c6bb7/volumes" Oct 31 01:07:39.458506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:39.458447 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:40.993913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:40.993875 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:42.657285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:42.657247 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:44.181216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:44.181178 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:45.774764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:45.774721 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:47.400969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:47.400930 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:48.323943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:48.323915 5978 scope.go:122] "RemoveContainer" containerID="80ff8de69a867e820aa42bd075e5ee464951f5b3d2e9f52f77b593e1c0810ee0" Oct 31 01:07:48.334231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:48.334212 5978 scope.go:122] "RemoveContainer" containerID="cfdbd0abb2a561899de3922e23a4fec6ed9049a797048900b5d898467be545fe" Oct 31 01:07:48.608393 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:48.608296 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:48.959677 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:48.959587 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:50.555223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:50.555184 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:51.937177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:51.937135 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:52.155016 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:52.154974 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:54.101348 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:54.101265 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:55.904576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:55.904525 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:57.451514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:57.450979 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:57.944015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:57.943981 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:58.509622 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:58.509434 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:59.070480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.055167 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:59.128510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.122558 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27"] Oct 31 01:07:59.128510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.122848 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" podUID="ea368ff9-fb95-48f5-99eb-c73749668dd6" containerName="maas-api" containerID="cri-o://5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d" gracePeriod=30 Oct 31 01:07:59.223415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.223244 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:59.443000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.442692 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:59.541475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.541393 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 01:07:59.543084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.543047 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea368ff9-fb95-48f5-99eb-c73749668dd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea368ff9-fb95-48f5-99eb-c73749668dd6-openshift-service-ca\") pod \"ea368ff9-fb95-48f5-99eb-c73749668dd6\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " Oct 31 01:07:59.543325 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.543295 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea368ff9-fb95-48f5-99eb-c73749668dd6-kube-api-access-k4rc2\" (UniqueName: \"kubernetes.io/projected/ea368ff9-fb95-48f5-99eb-c73749668dd6-kube-api-access-k4rc2\") pod \"ea368ff9-fb95-48f5-99eb-c73749668dd6\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " Oct 31 01:07:59.545565 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.545444 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea368ff9-fb95-48f5-99eb-c73749668dd6-openshift-service-ca" pod "ea368ff9-fb95-48f5-99eb-c73749668dd6" (UID: "ea368ff9-fb95-48f5-99eb-c73749668dd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:59.546255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.546240 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea368ff9-fb95-48f5-99eb-c73749668dd6-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:59.575247 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.575189 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea368ff9-fb95-48f5-99eb-c73749668dd6-kube-api-access-k4rc2" pod "ea368ff9-fb95-48f5-99eb-c73749668dd6" (UID: "ea368ff9-fb95-48f5-99eb-c73749668dd6"). InnerVolumeSpecName "kube-api-access-k4rc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:59.648707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.647543 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls\") pod \"ea368ff9-fb95-48f5-99eb-c73749668dd6\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " Oct 31 01:07:59.648707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.647607 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-metrics-certs\") pod \"ea368ff9-fb95-48f5-99eb-c73749668dd6\" (UID: \"ea368ff9-fb95-48f5-99eb-c73749668dd6\") " Oct 31 01:07:59.648707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.648226 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4rc2\" (UniqueName: \"kubernetes.io/projected/ea368ff9-fb95-48f5-99eb-c73749668dd6-kube-api-access-k4rc2\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:59.652515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.652483 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls" pod "ea368ff9-fb95-48f5-99eb-c73749668dd6" (UID: "ea368ff9-fb95-48f5-99eb-c73749668dd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:59.659888 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.659864 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-metrics-certs" pod "ea368ff9-fb95-48f5-99eb-c73749668dd6" (UID: "ea368ff9-fb95-48f5-99eb-c73749668dd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:59.712252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.712212 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:59.719142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.719111 5978 generic.go:356] "Generic (PLEG): container finished" podID="ea368ff9-fb95-48f5-99eb-c73749668dd6" containerID="5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d" exitCode=0 Oct 31 01:07:59.719300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.719164 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" Oct 31 01:07:59.719300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.719195 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" event={"ID":"ea368ff9-fb95-48f5-99eb-c73749668dd6","Type":"ContainerDied","Data":"5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d"} Oct 31 01:07:59.719300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.719253 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27" event={"ID":"ea368ff9-fb95-48f5-99eb-c73749668dd6","Type":"ContainerDied","Data":"f26e4a32c764894119a2d0b6eae79a1c2683846e44b29d1e8cf3aeb0010edab9"} Oct 31 01:07:59.719300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.719276 5978 scope.go:122] "RemoveContainer" containerID="5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d" Oct 31 01:07:59.732000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.731878 5978 scope.go:122] "RemoveContainer" containerID="5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d" Oct 31 01:07:59.732320 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:07:59.732269 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d\": container with ID starting with 5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d not found: ID does not exist" containerID="5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d" Oct 31 01:07:59.732441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.732331 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d"} err="failed to get container status \"5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d\": rpc error: code = NotFound desc = could not find container \"5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d\": container with ID starting with 5aa0093131fcaab0b1c39e4697c638fd0c29b662f336cc8912f1c3b4c9feb71d not found: ID does not exist" Oct 31 01:07:59.750384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.750350 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:59.750384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.750382 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea368ff9-fb95-48f5-99eb-c73749668dd6-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:07:59.770418 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.770384 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27"] Oct 31 01:07:59.777943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:07:59.777907 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4f4e69-5456985799-mcf27"] Oct 31 01:08:00.019240 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:00.019142 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:00.186501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:00.186435 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:00.353922 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:00.353889 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:00.428906 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:00.428867 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea368ff9-fb95-48f5-99eb-c73749668dd6" path="/var/lib/kubelet/pods/ea368ff9-fb95-48f5-99eb-c73749668dd6/volumes" Oct 31 01:08:01.790483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:01.790404 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:02.606435 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:02.606395 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k"] Oct 31 01:08:02.637204 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:02.637167 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk"] Oct 31 01:08:02.787904 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:02.787867 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:02.943078 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:02.942991 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:03.185625 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:08:03.185585 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:08:03.201129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:03.201029 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:03.353050 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:03.353016 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:03.730447 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:03.730404 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:04.900931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:04.900887 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:05.073147 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:05.073101 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:05.649973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:05.649929 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:06.716081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:06.716048 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:06.857086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:06.857049 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:07.001928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:07.001838 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t"] Oct 31 01:08:07.002111 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:07.002065 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" containerID="cri-o://d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917" gracePeriod=30 Oct 31 01:08:07.086004 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:07.085966 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:07.503272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:07.503232 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:07.607267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:07.607221 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" podUID="1e08c87e-bbc1-4fbd-8881-e00ebf387a78" containerName="payload-processing" containerID="cri-o://17b00932f135066f4c85fd19344ed4461ec691aaa9bc502b3873565f907edcbe" gracePeriod=25 Oct 31 01:08:07.637544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:07.637493 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" podUID="23d8fc09-02c3-4eff-a125-4b7d4966ef49" containerName="payload-pre-processing" containerID="cri-o://f900426fb48e2c2962a183f4eb36db310325dacf9333bb9e44182eeb9118078c" gracePeriod=25 Oct 31 01:08:09.055714 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:09.055677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:09.548019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:09.547977 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:09.679995 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:09.679960 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:10.220594 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:10.220552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:11.366709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:11.366674 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:12.262141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.262099 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" podUID="23d8fc09-02c3-4eff-a125-4b7d4966ef49" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:08:12.262336 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.262104 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" podUID="1e08c87e-bbc1-4fbd-8881-e00ebf387a78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:08:12.305277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.305247 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 01:08:12.390647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.390610 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-socket\") pod \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " Oct 31 01:08:12.391184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.390658 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-podinfo\") pod \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " Oct 31 01:08:12.391184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.390806 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-kube-api-access-wr8wm\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-kube-api-access-wr8wm\") pod \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " Oct 31 01:08:12.391184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.390858 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-envoy\") pod \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " Oct 31 01:08:12.391184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.390920 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-certs\") pod \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " Oct 31 01:08:12.391184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.390962 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istiod-ca-cert\") pod \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " Oct 31 01:08:12.391184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.390988 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-socket" pod "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" (UID: "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:12.391184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.391048 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-token\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-token\") pod \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " Oct 31 01:08:12.391184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.391092 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-data\") pod \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " Oct 31 01:08:12.391692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.391207 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-credential-socket\") pod \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\" (UID: \"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8\") " Oct 31 01:08:12.391692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.391276 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-certs" pod "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" (UID: "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:12.391692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.391497 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-data" pod "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" (UID: "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:12.391871 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.391700 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-credential-socket" pod "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" (UID: "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:12.391871 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.391851 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istiod-ca-cert" pod "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" (UID: "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:12.392055 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.392033 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:12.392127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.392069 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:12.392127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.392083 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:12.392127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.392095 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:12.392127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.392110 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:12.393924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.393894 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-envoy" pod "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" (UID: "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:12.394071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.394027 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-token" pod "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" (UID: "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:12.394333 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.394306 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-kube-api-access-wr8wm" pod "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" (UID: "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8"). InnerVolumeSpecName "kube-api-access-wr8wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:12.395534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.395507 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-podinfo" pod "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" (UID: "5bde62e5-02bf-4b4b-bfd7-4022e8616ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:08:12.493313 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.493278 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:12.493313 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.493310 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:12.493313 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.493320 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr8wm\" (UniqueName: \"kubernetes.io/projected/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-kube-api-access-wr8wm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:12.493620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.493333 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:12.863209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.863169 5978 generic.go:356] "Generic (PLEG): container finished" podID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerID="d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917" exitCode=0 Oct 31 01:08:12.863398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.863234 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" Oct 31 01:08:12.863398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.863258 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" event={"ID":"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8","Type":"ContainerDied","Data":"d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917"} Oct 31 01:08:12.863398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.863315 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t" event={"ID":"5bde62e5-02bf-4b4b-bfd7-4022e8616ab8","Type":"ContainerDied","Data":"b2e030545c514f9bac043d0a7c51158fbad81e9895c2d280d271de4776eaa90d"} Oct 31 01:08:12.863398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.863337 5978 scope.go:122] "RemoveContainer" containerID="d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917" Oct 31 01:08:12.874630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.874608 5978 scope.go:122] "RemoveContainer" containerID="bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5" Oct 31 01:08:12.885304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.885280 5978 scope.go:122] "RemoveContainer" containerID="d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917" Oct 31 01:08:12.885649 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:08:12.885628 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917\": container with ID starting with d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917 not found: ID does not exist" containerID="d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917" Oct 31 01:08:12.885714 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.885660 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917"} err="failed to get container status \"d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917\": rpc error: code = NotFound desc = could not find container \"d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917\": container with ID starting with d361c2db2d6eb4e759db92d5408eabd1ce6c7bed0fa16949fa87e3f9d3bd9917 not found: ID does not exist" Oct 31 01:08:12.885714 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.885680 5978 scope.go:122] "RemoveContainer" containerID="bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5" Oct 31 01:08:12.885965 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:08:12.885947 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5\": container with ID starting with bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5 not found: ID does not exist" containerID="bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5" Oct 31 01:08:12.886025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.885968 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5"} err="failed to get container status \"bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5\": rpc error: code = NotFound desc = could not find container \"bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5\": container with ID starting with bfb9c82cf2f8bbff062bb7218e6ad0ba7637622ee74d6d9c7d8440811b861df5 not found: ID does not exist" Oct 31 01:08:12.905113 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.905051 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:12.910630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.910565 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t"] Oct 31 01:08:12.920726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:12.920688 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4f4e69-openshift-default-58bcbfbb57-7q48t"] Oct 31 01:08:14.426794 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:14.426760 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" path="/var/lib/kubelet/pods/5bde62e5-02bf-4b4b-bfd7-4022e8616ab8/volumes" Oct 31 01:08:14.502349 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:14.498803 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:16.061542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:16.061508 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:16.429984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:16.429950 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:16.575121 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:16.575087 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:16.728362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:16.728265 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2"] Oct 31 01:08:16.728579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:16.728547 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" containerID="cri-o://e227f25fb86fb607f3c8531274dbbd2d771d5fd41fe617b8bb3f61309b314f36" gracePeriod=30 Oct 31 01:08:16.796126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:16.796094 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:17.809268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:17.809230 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:17.986483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:17.986418 5978 patch_prober.go:29] interesting pod/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:08:17.986687 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:17.986509 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:08:19.497127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:19.497088 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:21.233024 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:21.232944 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:21.969731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:21.969695 5978 generic.go:356] "Generic (PLEG): container finished" podID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerID="e227f25fb86fb607f3c8531274dbbd2d771d5fd41fe617b8bb3f61309b314f36" exitCode=0 Oct 31 01:08:21.969916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:21.969763 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" event={"ID":"5c26f4af-e8d0-4ad9-95f7-46d841598ea0","Type":"ContainerDied","Data":"e227f25fb86fb607f3c8531274dbbd2d771d5fd41fe617b8bb3f61309b314f36"} Oct 31 01:08:21.969916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:21.969829 5978 scope.go:122] "RemoveContainer" containerID="92bca105e6f7526d9a0c2115e34c2e70200172004fb9f3568c18b9f264d05e98" Oct 31 01:08:22.027699 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.027667 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 01:08:22.068569 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.068529 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-certs\") pod \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " Oct 31 01:08:22.068777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.068585 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-envoy\") pod \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " Oct 31 01:08:22.068777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.068627 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istiod-ca-cert\") pod \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " Oct 31 01:08:22.068777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.068691 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-data\") pod \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " Oct 31 01:08:22.068777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.068731 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-kube-api-access-flnrl\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-kube-api-access-flnrl\") pod \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " Oct 31 01:08:22.069017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.068838 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-certs" pod "5c26f4af-e8d0-4ad9-95f7-46d841598ea0" (UID: "5c26f4af-e8d0-4ad9-95f7-46d841598ea0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:22.069017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.068844 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-socket\") pod \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " Oct 31 01:08:22.069017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.068933 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-token\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-token\") pod \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " Oct 31 01:08:22.069195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069025 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-podinfo\") pod \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " Oct 31 01:08:22.069195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069033 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-data" pod "5c26f4af-e8d0-4ad9-95f7-46d841598ea0" (UID: "5c26f4af-e8d0-4ad9-95f7-46d841598ea0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:22.069195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069103 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-socket" pod "5c26f4af-e8d0-4ad9-95f7-46d841598ea0" (UID: "5c26f4af-e8d0-4ad9-95f7-46d841598ea0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:22.069195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069164 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istiod-ca-cert" pod "5c26f4af-e8d0-4ad9-95f7-46d841598ea0" (UID: "5c26f4af-e8d0-4ad9-95f7-46d841598ea0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:22.069397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069206 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-credential-socket\") pod \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\" (UID: \"5c26f4af-e8d0-4ad9-95f7-46d841598ea0\") " Oct 31 01:08:22.069497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069399 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-credential-socket" pod "5c26f4af-e8d0-4ad9-95f7-46d841598ea0" (UID: "5c26f4af-e8d0-4ad9-95f7-46d841598ea0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:22.069922 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069898 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:22.069992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069930 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:22.069992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069945 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:22.069992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069957 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:22.069992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.069970 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:22.071065 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.071042 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-envoy" pod "5c26f4af-e8d0-4ad9-95f7-46d841598ea0" (UID: "5c26f4af-e8d0-4ad9-95f7-46d841598ea0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:22.071189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.071113 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-kube-api-access-flnrl" pod "5c26f4af-e8d0-4ad9-95f7-46d841598ea0" (UID: "5c26f4af-e8d0-4ad9-95f7-46d841598ea0"). InnerVolumeSpecName "kube-api-access-flnrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:22.071531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.071511 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-token" pod "5c26f4af-e8d0-4ad9-95f7-46d841598ea0" (UID: "5c26f4af-e8d0-4ad9-95f7-46d841598ea0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:22.071607 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.071539 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-podinfo" pod "5c26f4af-e8d0-4ad9-95f7-46d841598ea0" (UID: "5c26f4af-e8d0-4ad9-95f7-46d841598ea0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:08:22.171735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.171701 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:22.171735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.171732 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:22.171735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.171741 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:22.171992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.171752 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnrl\" (UniqueName: \"kubernetes.io/projected/5c26f4af-e8d0-4ad9-95f7-46d841598ea0-kube-api-access-flnrl\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:22.261817 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.261725 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" podUID="1e08c87e-bbc1-4fbd-8881-e00ebf387a78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:08:22.261817 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.261768 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" podUID="23d8fc09-02c3-4eff-a125-4b7d4966ef49" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:08:22.978974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.978939 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" Oct 31 01:08:22.979187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.978966 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2" event={"ID":"5c26f4af-e8d0-4ad9-95f7-46d841598ea0","Type":"ContainerDied","Data":"2c6c2c71ed354d069f72e5044cc9a9edbf87404c3228bb1b8b85f41068514ab5"} Oct 31 01:08:22.979187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.979040 5978 scope.go:122] "RemoveContainer" containerID="e227f25fb86fb607f3c8531274dbbd2d771d5fd41fe617b8bb3f61309b314f36" Oct 31 01:08:22.995921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:22.995891 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:23.032565 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:23.032398 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2"] Oct 31 01:08:23.039319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:23.039281 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c942c9-openshift-default-5bdd576867-mjfz2"] Oct 31 01:08:24.429540 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:24.429506 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" path="/var/lib/kubelet/pods/5c26f4af-e8d0-4ad9-95f7-46d841598ea0/volumes" Oct 31 01:08:24.606695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:24.606656 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:25.166383 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.166338 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds"] Oct 31 01:08:25.167426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167396 5978 cpu_manager.go:414] "containerMap: removing container" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" Oct 31 01:08:25.167426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167422 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167439 5978 cpu_manager.go:414] "containerMap: removing container" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167446 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167496 5978 cpu_manager.go:414] "containerMap: removing container" podUID="a1b4397e-f00d-402e-9cb2-161b2e8396bd" containerName="maas-api" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167504 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b4397e-f00d-402e-9cb2-161b2e8396bd" containerName="maas-api" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167520 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8732d41a-8b1b-49dc-b421-1df166cf5a74" containerName="payload-processing" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167528 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8732d41a-8b1b-49dc-b421-1df166cf5a74" containerName="payload-processing" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167539 5978 cpu_manager.go:414] "containerMap: removing container" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167548 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167563 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167570 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167596 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167602 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167630 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167637 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167653 5978 cpu_manager.go:414] "containerMap: removing container" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167659 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167680 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5fe0e284-ca30-495d-8462-c29e540c6bb7" containerName="main" Oct 31 01:08:25.167688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167686 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe0e284-ca30-495d-8462-c29e540c6bb7" containerName="main" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167705 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ea368ff9-fb95-48f5-99eb-c73749668dd6" containerName="maas-api" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167712 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea368ff9-fb95-48f5-99eb-c73749668dd6" containerName="maas-api" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167731 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167738 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8d9746-73e2-43f0-bb4f-5b422fcb0ae7" containerName="istio-proxy" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167750 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167757 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167933 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe0e284-ca30-495d-8462-c29e540c6bb7" containerName="main" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167945 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea368ff9-fb95-48f5-99eb-c73749668dd6" containerName="maas-api" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167958 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167969 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167979 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da83d69-b959-4613-9b4d-d869ddb1884c" containerName="istio-proxy" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.167991 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b4397e-f00d-402e-9cb2-161b2e8396bd" containerName="maas-api" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.168002 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8732d41a-8b1b-49dc-b421-1df166cf5a74" containerName="payload-processing" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.168012 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.168022 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" Oct 31 01:08:25.168616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.168034 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bde62e5-02bf-4b4b-bfd7-4022e8616ab8" containerName="istio-proxy" Oct 31 01:08:25.173282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.173261 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.186611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.186560 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-328d61-openshift-default-dockercfg-5b75g\"" Oct 31 01:08:25.201075 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.201039 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds"] Oct 31 01:08:25.213038 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.212999 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-envoy\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.213238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.213083 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-credential-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.213238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.213118 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-token\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.213238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.213156 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zll\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-kube-api-access-v4zll\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.213238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.213221 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-podinfo\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.213560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.213258 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.213560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.213307 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-certs\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.213560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.213425 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b0ded-3105-4811-ad7d-ab09acd4cb30-istiod-ca-cert\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.213742 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.213582 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-data\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.314750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.314717 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-data\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.314935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.314794 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-envoy\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.314935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.314851 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-credential-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.314935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.314881 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-token\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.314935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.314912 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zll\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-kube-api-access-v4zll\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.315211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.314973 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-podinfo\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.315211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.315009 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.315211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.315062 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-certs\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.315211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.315178 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b0ded-3105-4811-ad7d-ab09acd4cb30-istiod-ca-cert\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.315211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.315179 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-data\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.315586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.315285 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-credential-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.315586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.315523 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-socket\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.315586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.315550 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-certs\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.315864 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.315846 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b0ded-3105-4811-ad7d-ab09acd4cb30-istiod-ca-cert\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.317695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.317665 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-envoy\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.318053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.318027 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-podinfo\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.335891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.335861 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-token\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.336099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.336076 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zll\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-kube-api-access-v4zll\") pod \"e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.488444 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.488358 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:25.807179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.807150 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds"] Oct 31 01:08:25.809502 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:08:25.809473 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147b0ded_3105_4811_ad7d_ab09acd4cb30.slice/crio-b87fd5cd6208188b6593d075c80bf0f7848e8d4139e973537c06f2279b8d5b83 WatchSource:0}: Error finding container b87fd5cd6208188b6593d075c80bf0f7848e8d4139e973537c06f2279b8d5b83: Status 404 returned error can't find the container with id b87fd5cd6208188b6593d075c80bf0f7848e8d4139e973537c06f2279b8d5b83 Oct 31 01:08:25.812401 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.812364 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:08:25.812511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.812441 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:08:25.812511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:25.812502 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:08:26.005320 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:26.005275 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" event={"ID":"147b0ded-3105-4811-ad7d-ab09acd4cb30","Type":"ContainerStarted","Data":"3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31"} Oct 31 01:08:26.005592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:26.005341 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" event={"ID":"147b0ded-3105-4811-ad7d-ab09acd4cb30","Type":"ContainerStarted","Data":"b87fd5cd6208188b6593d075c80bf0f7848e8d4139e973537c06f2279b8d5b83"} Oct 31 01:08:26.048805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:26.048755 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" podStartSLOduration=1.048738528 podStartE2EDuration="1.048738528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:26.046705814 +0000 UTC m=+2923.744819578" watchObservedRunningTime="2026-10-31 01:08:26.048738528 +0000 UTC m=+2923.746852292" Oct 31 01:08:26.188433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:26.188397 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:26.488700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:26.488605 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:26.490426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:26.490404 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 01:08:26.490559 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:26.490496 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 01:08:27.488769 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:27.488735 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 01:08:27.489192 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:27.488789 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 01:08:27.884264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:27.875791 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:28.042719 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.041721 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:28.058272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.058232 5978 generic.go:356] "Generic (PLEG): container finished" podID="1e08c87e-bbc1-4fbd-8881-e00ebf387a78" containerID="17b00932f135066f4c85fd19344ed4461ec691aaa9bc502b3873565f907edcbe" exitCode=0 Oct 31 01:08:28.058477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.058385 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" event={"ID":"1e08c87e-bbc1-4fbd-8881-e00ebf387a78","Type":"ContainerDied","Data":"17b00932f135066f4c85fd19344ed4461ec691aaa9bc502b3873565f907edcbe"} Oct 31 01:08:28.058477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.058442 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" event={"ID":"1e08c87e-bbc1-4fbd-8881-e00ebf387a78","Type":"ContainerDied","Data":"4b8d3b4bea65711176a5dc27a3dcc9be3967ca41a87d2f3b21d32fa8c4e8027f"} Oct 31 01:08:28.058625 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.058482 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8d3b4bea65711176a5dc27a3dcc9be3967ca41a87d2f3b21d32fa8c4e8027f" Oct 31 01:08:28.062159 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.062114 5978 generic.go:356] "Generic (PLEG): container finished" podID="23d8fc09-02c3-4eff-a125-4b7d4966ef49" containerID="f900426fb48e2c2962a183f4eb36db310325dacf9333bb9e44182eeb9118078c" exitCode=0 Oct 31 01:08:28.063573 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.063543 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" event={"ID":"23d8fc09-02c3-4eff-a125-4b7d4966ef49","Type":"ContainerDied","Data":"f900426fb48e2c2962a183f4eb36db310325dacf9333bb9e44182eeb9118078c"} Oct 31 01:08:28.108090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.108055 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 01:08:28.184058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.183992 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:28.190502 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.189111 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-plugins-config-volume\") pod \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\" (UID: \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\") " Oct 31 01:08:28.190502 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.189274 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-kube-api-access-czl95\" (UniqueName: \"kubernetes.io/projected/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-kube-api-access-czl95\") pod \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\" (UID: \"1e08c87e-bbc1-4fbd-8881-e00ebf387a78\") " Oct 31 01:08:28.191134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.191107 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-plugins-config-volume" pod "1e08c87e-bbc1-4fbd-8881-e00ebf387a78" (UID: "1e08c87e-bbc1-4fbd-8881-e00ebf387a78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:28.193350 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.193321 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-kube-api-access-czl95" pod "1e08c87e-bbc1-4fbd-8881-e00ebf387a78" (UID: "1e08c87e-bbc1-4fbd-8881-e00ebf387a78"). InnerVolumeSpecName "kube-api-access-czl95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:28.198309 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.198282 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 01:08:28.290970 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.290890 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d8fc09-02c3-4eff-a125-4b7d4966ef49-kube-api-access-f9kch\" (UniqueName: \"kubernetes.io/projected/23d8fc09-02c3-4eff-a125-4b7d4966ef49-kube-api-access-f9kch\") pod \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\" (UID: \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\") " Oct 31 01:08:28.291150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.291024 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d8fc09-02c3-4eff-a125-4b7d4966ef49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d8fc09-02c3-4eff-a125-4b7d4966ef49-plugins-config-volume\") pod \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\" (UID: \"23d8fc09-02c3-4eff-a125-4b7d4966ef49\") " Oct 31 01:08:28.291427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.291390 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d8fc09-02c3-4eff-a125-4b7d4966ef49-plugins-config-volume" pod "23d8fc09-02c3-4eff-a125-4b7d4966ef49" (UID: "23d8fc09-02c3-4eff-a125-4b7d4966ef49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:28.291597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.291487 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:28.291597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.291501 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czl95\" (UniqueName: \"kubernetes.io/projected/1e08c87e-bbc1-4fbd-8881-e00ebf387a78-kube-api-access-czl95\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:28.293385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.293361 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d8fc09-02c3-4eff-a125-4b7d4966ef49-kube-api-access-f9kch" pod "23d8fc09-02c3-4eff-a125-4b7d4966ef49" (UID: "23d8fc09-02c3-4eff-a125-4b7d4966ef49"). InnerVolumeSpecName "kube-api-access-f9kch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:28.393399 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.393362 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9kch\" (UniqueName: \"kubernetes.io/projected/23d8fc09-02c3-4eff-a125-4b7d4966ef49-kube-api-access-f9kch\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:28.393399 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.393392 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d8fc09-02c3-4eff-a125-4b7d4966ef49-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:08:28.489891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.489853 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 01:08:28.490578 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:28.490535 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 01:08:29.081532 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.081487 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" event={"ID":"23d8fc09-02c3-4eff-a125-4b7d4966ef49","Type":"ContainerDied","Data":"eb46a5ca5bc81a9e1c976377f10e13aa9907810bb3c95768066ee11100832cd0"} Oct 31 01:08:29.081532 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.081516 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk" Oct 31 01:08:29.081810 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.081585 5978 scope.go:122] "RemoveContainer" containerID="f900426fb48e2c2962a183f4eb36db310325dacf9333bb9e44182eeb9118078c" Oct 31 01:08:29.081810 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.081792 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k" Oct 31 01:08:29.138740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.138443 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k"] Oct 31 01:08:29.141834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.141802 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c942c9-68f5b49c89-pfr2k"] Oct 31 01:08:29.167923 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.167858 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk"] Oct 31 01:08:29.174840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.174807 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c942c9-6f77df795f-5mbwk"] Oct 31 01:08:29.452320 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.452229 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:29.489757 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.489725 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 01:08:29.489978 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:29.489802 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 01:08:30.103224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:30.103189 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:30.443972 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:30.443892 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e08c87e-bbc1-4fbd-8881-e00ebf387a78" path="/var/lib/kubelet/pods/1e08c87e-bbc1-4fbd-8881-e00ebf387a78/volumes" Oct 31 01:08:30.444352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:30.444338 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d8fc09-02c3-4eff-a125-4b7d4966ef49" path="/var/lib/kubelet/pods/23d8fc09-02c3-4eff-a125-4b7d4966ef49/volumes" Oct 31 01:08:30.489462 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:30.489426 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 01:08:30.489650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:30.489536 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 01:08:31.135586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:31.135549 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:31.489576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:31.489447 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 01:08:31.489576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:31.489555 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 01:08:32.489115 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:32.489072 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 31 01:08:32.489565 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:32.489161 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 31 01:08:32.822954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:32.822895 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:33.187616 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:08:33.187525 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:08:33.494983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:33.494899 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:33.495921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:33.495900 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:33.496963 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:33.496942 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:08:34.607541 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:34.607501 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:36.241239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:36.241202 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:37.706493 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.706432 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w"] Oct 31 01:08:37.707619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.707597 5978 cpu_manager.go:414] "containerMap: removing container" podUID="1e08c87e-bbc1-4fbd-8881-e00ebf387a78" containerName="payload-processing" Oct 31 01:08:37.707673 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.707622 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e08c87e-bbc1-4fbd-8881-e00ebf387a78" containerName="payload-processing" Oct 31 01:08:37.707673 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.707654 5978 cpu_manager.go:414] "containerMap: removing container" podUID="23d8fc09-02c3-4eff-a125-4b7d4966ef49" containerName="payload-pre-processing" Oct 31 01:08:37.707673 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.707662 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d8fc09-02c3-4eff-a125-4b7d4966ef49" containerName="payload-pre-processing" Oct 31 01:08:37.707894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.707876 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e08c87e-bbc1-4fbd-8881-e00ebf387a78" containerName="payload-processing" Oct 31 01:08:37.707932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.707898 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d8fc09-02c3-4eff-a125-4b7d4966ef49" containerName="payload-pre-processing" Oct 31 01:08:37.707932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.707911 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="65fa924f-1429-48ed-8ca2-bb5a3fca1f92" containerName="istio-proxy" Oct 31 01:08:37.707932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.707922 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c26f4af-e8d0-4ad9-95f7-46d841598ea0" containerName="istio-proxy" Oct 31 01:08:37.712960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.712940 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.729542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.729507 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ada0c1-serving-cert\"" Oct 31 01:08:37.745006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.744969 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w"] Oct 31 01:08:37.770315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.770265 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d84e9d-39fe-400a-9981-f4588885fa53-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.770569 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.770402 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls5r7\" (UniqueName: \"kubernetes.io/projected/41d84e9d-39fe-400a-9981-f4588885fa53-kube-api-access-ls5r7\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.770675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.770594 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.770812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.770789 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.821067 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.821034 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:37.879351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.879264 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.879812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.879472 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.879812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.879550 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d84e9d-39fe-400a-9981-f4588885fa53-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.879812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.879619 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls5r7\" (UniqueName: \"kubernetes.io/projected/41d84e9d-39fe-400a-9981-f4588885fa53-kube-api-access-ls5r7\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.881918 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:08:37.880711 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-serving-cert: secret "maas-api-e2e-shared-b-w3-ada0c1-serving-cert" not found Oct 31 01:08:37.881918 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:08:37.880821 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls podName:41d84e9d-39fe-400a-9981-f4588885fa53 nodeName:}" failed. No retries permitted until 2026-10-31 01:08:38.38079822 +0000 UTC m=+2936.078911966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls") pod "maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" (UID: "41d84e9d-39fe-400a-9981-f4588885fa53") : secret "maas-api-e2e-shared-b-w3-ada0c1-serving-cert" not found Oct 31 01:08:37.885869 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.881960 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d84e9d-39fe-400a-9981-f4588885fa53-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.903056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.895726 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.914429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.914168 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls5r7\" (UniqueName: \"kubernetes.io/projected/41d84e9d-39fe-400a-9981-f4588885fa53-kube-api-access-ls5r7\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:37.969423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:37.969341 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:38.140387 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:38.140349 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:38.386743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:38.386701 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:38.389135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:38.389108 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:38.632552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:38.632521 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:38.950748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:38.950716 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w"] Oct 31 01:08:38.952242 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:08:38.952210 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d84e9d_39fe_400a_9981_f4588885fa53.slice/crio-dc644444c8a3578573384053dec4e2997dcf0db5b542d36c154010e287fabb8a WatchSource:0}: Error finding container dc644444c8a3578573384053dec4e2997dcf0db5b542d36c154010e287fabb8a: Status 404 returned error can't find the container with id dc644444c8a3578573384053dec4e2997dcf0db5b542d36c154010e287fabb8a Oct 31 01:08:38.953694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:38.953673 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:08:39.158131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:39.158092 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" event={"ID":"41d84e9d-39fe-400a-9981-f4588885fa53","Type":"ContainerStarted","Data":"dc644444c8a3578573384053dec4e2997dcf0db5b542d36c154010e287fabb8a"} Oct 31 01:08:39.584186 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:39.584150 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:40.168471 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:40.168421 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" event={"ID":"41d84e9d-39fe-400a-9981-f4588885fa53","Type":"ContainerStarted","Data":"bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf"} Oct 31 01:08:40.168893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:40.168525 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:40.209262 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:40.209207 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" podStartSLOduration=2.854171928 podStartE2EDuration="3.209192381s" totalImagesPullingTime="355.020453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:40.205481495 +0000 UTC m=+2937.903595261" watchObservedRunningTime="2026-10-31 01:08:40.209192381 +0000 UTC m=+2937.907306146" Oct 31 01:08:41.274352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:41.274313 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:42.785763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:42.777042 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:43.276803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:43.276767 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:43.414848 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:43.414808 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:44.381550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:44.381508 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:45.896214 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:45.895223 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:46.183690 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:46.183618 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:08:47.525073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:47.525035 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:48.375189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:48.375151 5978 scope.go:122] "RemoveContainer" containerID="17b00932f135066f4c85fd19344ed4461ec691aaa9bc502b3873565f907edcbe" Oct 31 01:08:48.386252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:48.386229 5978 scope.go:122] "RemoveContainer" containerID="e58b4f2227009657106706dffeb291b3b5e5a4ca66f22ae8b10f41193afdaf6a" Oct 31 01:08:48.742783 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:48.742657 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:49.186558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:49.186524 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:50.815961 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:50.815921 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:51.922869 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:51.922831 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:52.405582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:52.401735 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:54.211253 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:54.211215 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:55.531851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:55.531811 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:56.055510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:56.055448 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:56.231138 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:56.231102 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:56.377890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:56.377846 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:56.762435 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:56.762338 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:57.975224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:57.975059 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:59.606849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:08:59.606815 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:01.287148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:01.287112 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:02.908804 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:02.908763 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:03.185410 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:09:03.185313 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:09:03.274686 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:03.274647 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:03.430893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:03.430859 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:03.611542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:03.611508 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt"] Oct 31 01:09:03.611850 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:03.611817 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" containerID="cri-o://232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330" gracePeriod=30 Oct 31 01:09:03.699340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:03.699299 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:04.184612 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:04.184576 5978 patch_prober.go:29] interesting pod/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:09:04.185014 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:04.184639 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:09:04.487609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:04.487372 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:04.684344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:04.684301 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:05.214787 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:05.214749 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:05.383476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:05.370416 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.164693 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:06.162837 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:07.899748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:07.899711 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:08.884788 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:08.884727 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 01:09:09.078745 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.078710 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-certs\") pod \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " Oct 31 01:09:09.079243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.078796 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-kube-api-access-d944j\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-kube-api-access-d944j\") pod \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " Oct 31 01:09:09.079243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079028 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-certs" pod "3068a265-cfd5-41b1-b2ed-ddac0d8553ab" (UID: "3068a265-cfd5-41b1-b2ed-ddac0d8553ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:09.079243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079062 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-socket\") pod \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " Oct 31 01:09:09.079243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079163 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-podinfo\") pod \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " Oct 31 01:09:09.079243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079195 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-credential-socket\") pod \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " Oct 31 01:09:09.079243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079223 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istiod-ca-cert\") pod \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " Oct 31 01:09:09.079640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079250 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-envoy\") pod \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " Oct 31 01:09:09.079640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079278 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-socket" pod "3068a265-cfd5-41b1-b2ed-ddac0d8553ab" (UID: "3068a265-cfd5-41b1-b2ed-ddac0d8553ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:09.079640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079312 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-token\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-token\") pod \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " Oct 31 01:09:09.079640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079391 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-data\") pod \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\" (UID: \"3068a265-cfd5-41b1-b2ed-ddac0d8553ab\") " Oct 31 01:09:09.079640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.079464 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-credential-socket" pod "3068a265-cfd5-41b1-b2ed-ddac0d8553ab" (UID: "3068a265-cfd5-41b1-b2ed-ddac0d8553ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:09.080131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.080103 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:09:09.080131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.080128 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:09:09.080281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.080141 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:09:09.080281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.080161 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istiod-ca-cert" pod "3068a265-cfd5-41b1-b2ed-ddac0d8553ab" (UID: "3068a265-cfd5-41b1-b2ed-ddac0d8553ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:09.080281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.080225 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-data" pod "3068a265-cfd5-41b1-b2ed-ddac0d8553ab" (UID: "3068a265-cfd5-41b1-b2ed-ddac0d8553ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:09.081969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.081938 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-kube-api-access-d944j" pod "3068a265-cfd5-41b1-b2ed-ddac0d8553ab" (UID: "3068a265-cfd5-41b1-b2ed-ddac0d8553ab"). InnerVolumeSpecName "kube-api-access-d944j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:09.082227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.082200 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-envoy" pod "3068a265-cfd5-41b1-b2ed-ddac0d8553ab" (UID: "3068a265-cfd5-41b1-b2ed-ddac0d8553ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:09.082356 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.082314 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-token" pod "3068a265-cfd5-41b1-b2ed-ddac0d8553ab" (UID: "3068a265-cfd5-41b1-b2ed-ddac0d8553ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:09.082356 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.082350 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-podinfo" pod "3068a265-cfd5-41b1-b2ed-ddac0d8553ab" (UID: "3068a265-cfd5-41b1-b2ed-ddac0d8553ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:09:09.116048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.116011 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:09.181289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.181209 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:09:09.181289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.181240 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:09:09.181289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.181251 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:09:09.181289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.181259 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:09:09.181289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.181269 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:09:09.181289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.181278 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d944j\" (UniqueName: \"kubernetes.io/projected/3068a265-cfd5-41b1-b2ed-ddac0d8553ab-kube-api-access-d944j\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:09:09.258124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.258090 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:09.420417 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.420376 5978 generic.go:356] "Generic (PLEG): container finished" podID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerID="232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330" exitCode=0 Oct 31 01:09:09.420619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.420423 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" Oct 31 01:09:09.420619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.420464 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" event={"ID":"3068a265-cfd5-41b1-b2ed-ddac0d8553ab","Type":"ContainerDied","Data":"232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330"} Oct 31 01:09:09.420619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.420517 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt" event={"ID":"3068a265-cfd5-41b1-b2ed-ddac0d8553ab","Type":"ContainerDied","Data":"949d05e3434cc74ece77d0ec4d62b54ef2efb9d795cae7ecac2bf3b348d57b8b"} Oct 31 01:09:09.420619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.420540 5978 scope.go:122] "RemoveContainer" containerID="232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330" Oct 31 01:09:09.432192 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.432175 5978 scope.go:122] "RemoveContainer" containerID="1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d" Oct 31 01:09:09.442216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.442198 5978 scope.go:122] "RemoveContainer" containerID="232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330" Oct 31 01:09:09.442536 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:09:09.442503 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330\": container with ID starting with 232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330 not found: ID does not exist" containerID="232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330" Oct 31 01:09:09.442613 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.442544 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330"} err="failed to get container status \"232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330\": rpc error: code = NotFound desc = could not find container \"232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330\": container with ID starting with 232f0df1bb85a0b7b735b2566b11ff918b6dbf9a9474d3ebdb367d652be99330 not found: ID does not exist" Oct 31 01:09:09.442613 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.442563 5978 scope.go:122] "RemoveContainer" containerID="1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d" Oct 31 01:09:09.442823 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:09:09.442805 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d\": container with ID starting with 1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d not found: ID does not exist" containerID="1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d" Oct 31 01:09:09.442861 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.442832 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d"} err="failed to get container status \"1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d\": rpc error: code = NotFound desc = could not find container \"1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d\": container with ID starting with 1059cdbdbc0d1d2f3092aba8c85769d441b80b1800558c030f9cd9cc96c8c56d not found: ID does not exist" Oct 31 01:09:09.477364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.477328 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt"] Oct 31 01:09:09.486434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.486406 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6f8856-openshift-default-6f9ff5d847-kkklt"] Oct 31 01:09:09.654811 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:09.654773 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:10.427774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:10.427742 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" path="/var/lib/kubelet/pods/3068a265-cfd5-41b1-b2ed-ddac0d8553ab/volumes" Oct 31 01:09:10.774226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:10.774132 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:10.935226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:10.935189 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.034539 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.034378 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg"] Oct 31 01:09:11.036205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.036168 5978 cpu_manager.go:414] "containerMap: removing container" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" Oct 31 01:09:11.036205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.036203 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" Oct 31 01:09:11.036528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.036214 5978 cpu_manager.go:414] "containerMap: removing container" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" Oct 31 01:09:11.036528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.036222 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" Oct 31 01:09:11.036528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.036483 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" Oct 31 01:09:11.036528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.036505 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="3068a265-cfd5-41b1-b2ed-ddac0d8553ab" containerName="istio-proxy" Oct 31 01:09:11.045233 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.042684 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:11.065654 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.064508 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73f915\"" Oct 31 01:09:11.077505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.073636 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73f915-dockercfg-92zcl\"" Oct 31 01:09:11.084773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.082394 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg"] Oct 31 01:09:11.108542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.103614 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47gth\" (UniqueName: \"kubernetes.io/projected/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-kube-api-access-47gth\") pod \"payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg\" (UID: \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:11.108542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.103687 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg\" (UID: \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:11.205011 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.204974 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47gth\" (UniqueName: \"kubernetes.io/projected/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-kube-api-access-47gth\") pod \"payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg\" (UID: \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:11.205223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.205029 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg\" (UID: \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:11.205836 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.205810 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg\" (UID: \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:11.225997 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.225959 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47gth\" (UniqueName: \"kubernetes.io/projected/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-kube-api-access-47gth\") pod \"payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg\" (UID: \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:11.402303 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.402270 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:11.500985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.500892 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.702127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:11.702094 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg"] Oct 31 01:09:11.704030 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:09:11.704001 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ca3160_bfb1_4fd6_9a28_d5bdb1d2fec3.slice/crio-a1a2d1d76b781becb0c00d0e9fb526688fd48864b8666ce1e0e0db4daa6399b5 WatchSource:0}: Error finding container a1a2d1d76b781becb0c00d0e9fb526688fd48864b8666ce1e0e0db4daa6399b5: Status 404 returned error can't find the container with id a1a2d1d76b781becb0c00d0e9fb526688fd48864b8666ce1e0e0db4daa6399b5 Oct 31 01:09:12.446609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:12.446566 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" event={"ID":"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3","Type":"ContainerStarted","Data":"643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a"} Oct 31 01:09:12.446803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:12.446638 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" event={"ID":"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3","Type":"ContainerStarted","Data":"a1a2d1d76b781becb0c00d0e9fb526688fd48864b8666ce1e0e0db4daa6399b5"} Oct 31 01:09:12.446803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:12.446668 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:12.485941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:12.485866 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" podStartSLOduration=1.48584705 podStartE2EDuration="1.48584705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:12.482932065 +0000 UTC m=+2970.181045825" watchObservedRunningTime="2026-10-31 01:09:12.48584705 +0000 UTC m=+2970.183960815" Oct 31 01:09:13.134043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:13.127811 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.708543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:14.708501 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.885679 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:14.885623 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:16.393310 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:16.393270 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:17.994542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:17.994382 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:19.659872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:19.659831 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:21.185107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:21.185059 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:22.804924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:22.804884 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:23.457692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:23.457659 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:09:24.365528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:24.365494 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:26.116325 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:26.116286 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:27.755477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:27.754425 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:29.349795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:29.349757 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:30.907958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:30.907918 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:31.589123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:31.589086 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:32.627593 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:32.627560 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:33.182431 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:09:33.182392 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:09:34.287305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:34.287257 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:35.838111 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:35.838061 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:37.411814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:37.411743 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:39.083434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:39.083375 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:40.527729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.527692 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt"] Oct 31 01:09:40.532738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.532715 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.550726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.550690 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-73f915-serving-cert\"" Oct 31 01:09:40.561117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.561081 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt"] Oct 31 01:09:40.647286 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.647245 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:40.685315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.685279 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c009cbcf-83a3-4d7d-bcb0-205189531fdb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.685315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.685318 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.685609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.685438 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ws72\" (UniqueName: \"kubernetes.io/projected/c009cbcf-83a3-4d7d-bcb0-205189531fdb-kube-api-access-6ws72\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.685609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.685514 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-metrics-certs\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.787019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.786927 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c009cbcf-83a3-4d7d-bcb0-205189531fdb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.787019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.786973 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.787255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.787023 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ws72\" (UniqueName: \"kubernetes.io/projected/c009cbcf-83a3-4d7d-bcb0-205189531fdb-kube-api-access-6ws72\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.787255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.787043 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-metrics-certs\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.787255 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:09:40.787151 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-serving-cert: secret "maas-api-e2e-ipp-a-73f915-serving-cert" not found Oct 31 01:09:40.787255 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:09:40.787238 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls podName:c009cbcf-83a3-4d7d-bcb0-205189531fdb nodeName:}" failed. No retries permitted until 2026-10-31 01:09:41.287213355 +0000 UTC m=+2998.985327118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls") pod "maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" (UID: "c009cbcf-83a3-4d7d-bcb0-205189531fdb") : secret "maas-api-e2e-ipp-a-73f915-serving-cert" not found Oct 31 01:09:40.787696 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.787679 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c009cbcf-83a3-4d7d-bcb0-205189531fdb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.789827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.789800 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-metrics-certs\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.790397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.790374 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:40.810890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.810857 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ws72\" (UniqueName: \"kubernetes.io/projected/c009cbcf-83a3-4d7d-bcb0-205189531fdb-kube-api-access-6ws72\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:40.965858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:40.965816 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:41.295702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:41.295665 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:41.298340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:41.298314 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73f915-6c664865-h8lwt\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:41.449582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:41.449552 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:41.754402 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:41.754372 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt"] Oct 31 01:09:41.756312 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:09:41.756286 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc009cbcf_83a3_4d7d_bcb0_205189531fdb.slice/crio-66434dd48014af09ee8d008a33a7b68ead8398be7c99d9f7a83fa6ecde8032fa WatchSource:0}: Error finding container 66434dd48014af09ee8d008a33a7b68ead8398be7c99d9f7a83fa6ecde8032fa: Status 404 returned error can't find the container with id 66434dd48014af09ee8d008a33a7b68ead8398be7c99d9f7a83fa6ecde8032fa Oct 31 01:09:42.270701 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:42.270652 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:42.677174 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:42.677136 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" event={"ID":"c009cbcf-83a3-4d7d-bcb0-205189531fdb","Type":"ContainerStarted","Data":"ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561"} Oct 31 01:09:42.677373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:42.677202 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" event={"ID":"c009cbcf-83a3-4d7d-bcb0-205189531fdb","Type":"ContainerStarted","Data":"66434dd48014af09ee8d008a33a7b68ead8398be7c99d9f7a83fa6ecde8032fa"} Oct 31 01:09:42.677373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:42.677354 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:42.716427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:42.716371 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" podStartSLOduration=2.258773673 podStartE2EDuration="2.71635588s" totalImagesPullingTime="457.582207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:42.7149198 +0000 UTC m=+3000.413033566" watchObservedRunningTime="2026-10-31 01:09:42.71635588 +0000 UTC m=+3000.414469644" Oct 31 01:09:43.905707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:43.905664 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:45.603377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:45.602089 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:47.208752 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:47.208706 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:48.690898 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:48.690864 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:09:48.885839 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:48.885800 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:49.077545 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:49.077507 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:50.544342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:50.544315 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:09:50.547922 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:50.547899 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:09:50.809629 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:50.806664 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:52.528036 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:52.527930 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:54.142601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:54.142566 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:55.818735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:55.818692 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:57.426550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:57.426504 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:58.637511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:58.637468 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:59.086051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:59.086003 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:59.616200 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:09:59.616160 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:00.805932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:00.805878 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:02.485422 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:02.485386 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:03.182903 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:10:03.182865 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:10:04.031751 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:04.031704 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:05.613546 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:05.613398 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:07.198784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:07.198743 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:08.710056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:08.710018 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:10.319774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:10.319734 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:11.983189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:11.983152 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:13.550037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:13.549998 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:15.205475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:15.200431 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:17.130471 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:17.130147 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:19.089823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:19.089784 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:20.908430 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:20.908390 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:20.968323 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:20.968286 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg"] Oct 31 01:10:21.172875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:21.172776 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:21.341431 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:21.341388 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:22.583028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:22.582990 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:24.243216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:24.232072 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:25.905081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:25.905044 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:25.968625 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:25.968574 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" podUID="04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3" containerName="payload-processing" containerID="cri-o://643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a" gracePeriod=25 Oct 31 01:10:26.259123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:26.259093 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:10:26.431878 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:26.431848 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-kube-api-access-47gth\" (UniqueName: \"kubernetes.io/projected/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-kube-api-access-47gth\") pod \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\" (UID: \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\") " Oct 31 01:10:26.433661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:26.433624 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-plugins-config-volume\") pod \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\" (UID: \"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3\") " Oct 31 01:10:26.434050 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:26.434022 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-plugins-config-volume" pod "04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3" (UID: "04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:26.434157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:26.434035 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-kube-api-access-47gth" pod "04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3" (UID: "04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3"). InnerVolumeSpecName "kube-api-access-47gth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:26.434373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:26.434339 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47gth\" (UniqueName: \"kubernetes.io/projected/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-kube-api-access-47gth\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:10:26.434470 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:26.434380 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:10:27.048140 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.048103 5978 generic.go:356] "Generic (PLEG): container finished" podID="04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3" containerID="643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a" exitCode=0 Oct 31 01:10:27.048653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.048156 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" Oct 31 01:10:27.048653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.048189 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" event={"ID":"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3","Type":"ContainerDied","Data":"643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a"} Oct 31 01:10:27.048653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.048247 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg" event={"ID":"04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3","Type":"ContainerDied","Data":"a1a2d1d76b781becb0c00d0e9fb526688fd48864b8666ce1e0e0db4daa6399b5"} Oct 31 01:10:27.048653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.048275 5978 scope.go:122] "RemoveContainer" containerID="643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a" Oct 31 01:10:27.060903 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.060882 5978 scope.go:122] "RemoveContainer" containerID="643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a" Oct 31 01:10:27.061210 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:10:27.061186 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a\": container with ID starting with 643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a not found: ID does not exist" containerID="643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a" Oct 31 01:10:27.061262 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.061219 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a"} err="failed to get container status \"643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a\": rpc error: code = NotFound desc = could not find container \"643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a\": container with ID starting with 643119b811e1a0c30333140e63dc464e57946a00b050dfcea7e689a4ac2e2e9a not found: ID does not exist" Oct 31 01:10:27.094389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.094356 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg"] Oct 31 01:10:27.103612 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.103585 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f915-78fccdbd7f-98rsg"] Oct 31 01:10:27.596478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:27.596043 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:28.428122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:28.428087 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3" path="/var/lib/kubelet/pods/04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3/volumes" Oct 31 01:10:29.250107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:29.250071 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:30.898678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:30.898639 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:32.527313 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:32.527264 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:33.183063 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:10:33.183030 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:10:34.070025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:34.069981 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:35.396217 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:35.396178 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:35.540611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:35.540562 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:35.823911 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:35.823870 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:37.486325 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:37.486286 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:39.127986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:39.127821 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:40.720827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:40.720789 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:42.221181 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:42.221145 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:43.846405 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:43.846363 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:45.509952 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:45.509891 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:47.223972 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:47.223927 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:48.459920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:48.459889 5978 scope.go:122] "RemoveContainer" containerID="41487c71d750d75c6f1e458a7b6e97c76d5e37691d8cd755758917ecc0f304f0" Oct 31 01:10:48.802496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:48.802378 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:49.000340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:49.000288 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:50.159043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:50.159007 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:50.314159 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:50.314122 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:50.475315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:50.475228 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:51.675974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:51.675929 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:51.830495 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:51.827168 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:52.060522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.047679 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st"] Oct 31 01:10:52.060522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.056669 5978 cpu_manager.go:414] "containerMap: removing container" podUID="04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3" containerName="payload-processing" Oct 31 01:10:52.060522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.056696 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3" containerName="payload-processing" Oct 31 01:10:52.060522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.056876 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ca3160-bfb1-4fd6-9a28-d5bdb1d2fec3" containerName="payload-processing" Oct 31 01:10:52.063172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.063142 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:10:52.070828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.070794 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st"] Oct 31 01:10:52.084109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.084077 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73f915\"" Oct 31 01:10:52.093308 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.093278 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73f915-dockercfg-lklrm\"" Oct 31 01:10:52.147122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.147084 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st\" (UID: \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:10:52.147314 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.147222 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bkfq\" (UniqueName: \"kubernetes.io/projected/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-kube-api-access-5bkfq\") pod \"payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st\" (UID: \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:10:52.249164 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.249109 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bkfq\" (UniqueName: \"kubernetes.io/projected/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-kube-api-access-5bkfq\") pod \"payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st\" (UID: \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:10:52.249492 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.249438 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st\" (UID: \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:10:52.250468 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.250421 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st\" (UID: \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:10:52.270601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.270567 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bkfq\" (UniqueName: \"kubernetes.io/projected/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-kube-api-access-5bkfq\") pod \"payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st\" (UID: \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:10:52.395207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.395170 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:10:52.698233 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.698191 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st"] Oct 31 01:10:52.700575 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:10:52.700546 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf12e68_26b5_4104_82f2_a7f7d41fdb1e.slice/crio-dc2cbb70e38b681fe50a034839f23a060d0e723456c4c3c506cd33131d097505 WatchSource:0}: Error finding container dc2cbb70e38b681fe50a034839f23a060d0e723456c4c3c506cd33131d097505: Status 404 returned error can't find the container with id dc2cbb70e38b681fe50a034839f23a060d0e723456c4c3c506cd33131d097505 Oct 31 01:10:52.823179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:52.823084 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:53.276104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:53.276064 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" event={"ID":"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e","Type":"ContainerStarted","Data":"dc2cbb70e38b681fe50a034839f23a060d0e723456c4c3c506cd33131d097505"} Oct 31 01:10:54.390853 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:54.390825 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:55.293516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:55.293481 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" event={"ID":"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e","Type":"ContainerStarted","Data":"3faaa4e073313ae8c9441fd4cb9dbfc36f2a70d6c2f77b6c8d26a35542b8e752"} Oct 31 01:10:55.293734 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:55.293566 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:10:55.329807 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:55.329746 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" podStartSLOduration=1.674342868 podStartE2EDuration="3.329732695s" totalImagesPullingTime="1.655389827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:55.32951551 +0000 UTC m=+3073.027629275" watchObservedRunningTime="2026-10-31 01:10:55.329732695 +0000 UTC m=+3073.027846476" Oct 31 01:10:56.011099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:56.011054 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:57.615248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:57.615203 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:59.214602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:10:59.214559 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:00.730317 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:00.730276 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:02.376493 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:02.376431 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:03.182792 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:11:03.182750 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:11:04.031476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:04.030838 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:05.697736 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:05.697698 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:06.303745 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:06.303711 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:11:07.288301 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:07.288266 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:08.704221 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:08.704181 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:08.882727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:08.882689 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:09.327038 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.327000 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm"] Oct 31 01:11:09.332480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.332440 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.347104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.347061 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-75ead8-openshift-default-dockercfg-sdhw5\"" Oct 31 01:11:09.362888 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.362850 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm"] Oct 31 01:11:09.408999 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.408960 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-credential-socket\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.409201 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.409109 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-istio-token\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.409201 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.409162 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975j7\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-kube-api-access-975j7\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.409329 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.409241 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-socket\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.409400 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.409325 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-data\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.409400 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.409370 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4994c0aa-deec-4f24-8422-e5be874c350c-istiod-ca-cert\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.409498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.409483 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-certs\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.409576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.409562 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-envoy\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.409623 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.409601 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4994c0aa-deec-4f24-8422-e5be874c350c-istio-podinfo\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.431996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.431960 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:09.511246 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511205 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-credential-socket\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511308 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-istio-token\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511345 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975j7\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-kube-api-access-975j7\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511517 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-socket\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511592 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-data\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511632 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4994c0aa-deec-4f24-8422-e5be874c350c-istiod-ca-cert\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511832 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511710 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-certs\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511832 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511741 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-credential-socket\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511945 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511828 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-envoy\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511945 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511892 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4994c0aa-deec-4f24-8422-e5be874c350c-istio-podinfo\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.511945 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.511912 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-socket\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.512094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.512073 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-certs\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.512158 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.512133 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-data\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.512270 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.512250 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4994c0aa-deec-4f24-8422-e5be874c350c-istiod-ca-cert\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.514245 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.514222 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-envoy\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.514505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.514483 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4994c0aa-deec-4f24-8422-e5be874c350c-istio-podinfo\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.556632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.556556 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-istio-token\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.561309 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.559396 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-975j7\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-kube-api-access-975j7\") pod \"e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.580511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.580189 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:09.650238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.650200 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:09.968142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.968115 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm"] Oct 31 01:11:09.968831 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:11:09.968809 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4994c0aa_deec_4f24_8422_e5be874c350c.slice/crio-d00bd8568004f63831338f0205abf1f52484fd7461e3ab7bac2db74786c2e1c0 WatchSource:0}: Error finding container d00bd8568004f63831338f0205abf1f52484fd7461e3ab7bac2db74786c2e1c0: Status 404 returned error can't find the container with id d00bd8568004f63831338f0205abf1f52484fd7461e3ab7bac2db74786c2e1c0 Oct 31 01:11:09.971736 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.971702 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:09.971829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.971772 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:09.971829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:09.971804 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:11:10.426864 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:10.426830 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" event={"ID":"4994c0aa-deec-4f24-8422-e5be874c350c","Type":"ContainerStarted","Data":"c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf"} Oct 31 01:11:10.427053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:10.426885 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" event={"ID":"4994c0aa-deec-4f24-8422-e5be874c350c","Type":"ContainerStarted","Data":"d00bd8568004f63831338f0205abf1f52484fd7461e3ab7bac2db74786c2e1c0"} Oct 31 01:11:10.470160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:10.470106 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" podStartSLOduration=1.470089852 podStartE2EDuration="1.470089852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:10.469231991 +0000 UTC m=+3088.167345755" watchObservedRunningTime="2026-10-31 01:11:10.470089852 +0000 UTC m=+3088.168203616" Oct 31 01:11:10.597764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:10.597726 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:10.654965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:10.654395 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:11.657484 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:11.657444 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:12.215049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:12.215004 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:12.434335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:12.434304 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:12.435266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:12.435248 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:11:13.004255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:13.004220 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:13.160282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:13.160249 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:14.018029 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:14.017988 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:14.687767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:14.687468 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:14.825814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:14.825773 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:15.731049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:15.731013 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:17.519672 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:17.519603 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:19.236294 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:19.236261 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:20.857164 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:20.857127 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:22.322712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:22.322662 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:22.469342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:22.469303 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:24.051698 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:24.051662 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:24.200098 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:24.200065 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:25.718362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:25.718326 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:27.716019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:27.715983 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:29.434833 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:29.425967 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:31.063822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:31.063786 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:32.614568 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:32.614326 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:33.182538 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:11:33.182492 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:11:34.214909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:34.214855 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:35.903495 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:35.903436 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:37.505803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:37.505759 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:39.128371 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:39.128331 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:39.874403 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:39.874362 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:40.435309 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:40.435272 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:40.952405 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:40.952371 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:42.541083 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:42.541039 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:44.205548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:44.205510 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:45.883996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:45.883961 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:47.258336 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.258296 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8"] Oct 31 01:11:47.263271 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.263242 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.280643 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.280613 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-75ead8-serving-cert\"" Oct 31 01:11:47.286058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.286028 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8"] Oct 31 01:11:47.336712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.336675 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.336898 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.336835 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.336960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.336902 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn2zw\" (UniqueName: \"kubernetes.io/projected/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-kube-api-access-mn2zw\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.337006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.336967 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-metrics-certs\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.364910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.364868 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:47.438012 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.437972 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.438245 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.438132 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.438245 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.438219 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn2zw\" (UniqueName: \"kubernetes.io/projected/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-kube-api-access-mn2zw\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.438380 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.438267 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-metrics-certs\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.447971 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:11:47.447940 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-serving-cert: secret "maas-api-e2e-ipp-b-75ead8-serving-cert" not found Oct 31 01:11:47.448269 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:11:47.448256 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls podName:8ec2edd6-67f1-4bd4-b4ab-b64a389ead00 nodeName:}" failed. No retries permitted until 2026-10-31 01:11:47.948232532 +0000 UTC m=+3125.646346292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls") pod "maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" (UID: "8ec2edd6-67f1-4bd4-b4ab-b64a389ead00") : secret "maas-api-e2e-ipp-b-75ead8-serving-cert" not found Oct 31 01:11:47.449334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.449286 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.449932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.449906 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-metrics-certs\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.471446 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.471411 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn2zw\" (UniqueName: \"kubernetes.io/projected/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-kube-api-access-mn2zw\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:47.499372 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.499326 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:47.822632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:47.822587 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:48.047829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:48.047793 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:48.050203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:48.050180 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:48.180327 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:48.180245 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:48.487034 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:48.487003 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8"] Oct 31 01:11:48.489328 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:11:48.489298 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec2edd6_67f1_4bd4_b4ab_b64a389ead00.slice/crio-a89d9cc17e1c1eefb497d406d1ed7ca688626d90ffca56f60f04ccbc1d1fae41 WatchSource:0}: Error finding container a89d9cc17e1c1eefb497d406d1ed7ca688626d90ffca56f60f04ccbc1d1fae41: Status 404 returned error can't find the container with id a89d9cc17e1c1eefb497d406d1ed7ca688626d90ffca56f60f04ccbc1d1fae41 Oct 31 01:11:48.699771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:48.699730 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" event={"ID":"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00","Type":"ContainerStarted","Data":"a89d9cc17e1c1eefb497d406d1ed7ca688626d90ffca56f60f04ccbc1d1fae41"} Oct 31 01:11:49.078599 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:49.078557 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:49.222375 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:49.222268 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:49.366843 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:49.366805 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:49.708887 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:49.708853 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" event={"ID":"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00","Type":"ContainerStarted","Data":"175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3"} Oct 31 01:11:49.709305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:49.708922 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:49.747495 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:49.747420 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" podStartSLOduration=2.302137352 podStartE2EDuration="2.747404221s" totalImagesPullingTime="445.266869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:49.745677648 +0000 UTC m=+3127.443791412" watchObservedRunningTime="2026-10-31 01:11:49.747404221 +0000 UTC m=+3127.445517986" Oct 31 01:11:50.724267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:50.724223 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:52.279721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:52.279682 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:53.850405 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:53.850367 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:55.539283 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:55.539181 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:55.735588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:55.735556 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:11:57.206623 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:57.206582 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:58.911223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:11:58.911185 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:00.082852 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:00.082814 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:00.603257 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:00.603217 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:00.819992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:00.819945 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:02.423823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:02.423788 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:03.183508 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:12:03.183468 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:12:04.739699 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:04.739652 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:06.550355 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:06.550318 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:08.102819 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:08.101123 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:09.097609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:09.097572 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:09.933307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:09.933268 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:11.528603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:11.528544 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:13.023873 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:13.023828 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:14.674679 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:14.674641 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:16.316768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:16.316724 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:18.036921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:18.036886 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:19.688774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:19.688737 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:21.321738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:21.321695 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:21.843756 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:21.843722 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:22.475717 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:22.475673 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:22.985837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:22.985799 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:23.149617 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:23.149569 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:24.607801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:24.607706 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:26.320048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:26.319941 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:26.713516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:26.713419 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:26.861754 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:26.861711 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:28.204216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:28.204166 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:28.407943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:28.407903 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:28.565311 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:28.565216 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:29.888255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:29.888224 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:30.046382 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:30.046346 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:30.193761 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:30.193660 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:31.803588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:31.803548 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:32.349764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:32.349733 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:33.182665 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:12:33.182623 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:12:33.447758 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:33.447657 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:34.686470 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:34.686420 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:35.020327 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:35.020227 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:36.695573 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:36.695536 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:38.263513 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:38.262562 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:40.036056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:40.036010 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:42.235534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:42.235496 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:42.508021 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:42.507923 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:43.920678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:43.920636 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:45.422550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:45.422503 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:46.893259 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:46.893218 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:48.522129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:48.522077 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:49.261990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:49.261950 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:50.023580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:50.023539 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:51.583976 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:51.583941 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:53.177680 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:53.177613 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:54.867190 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:54.867153 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:56.400637 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:56.400590 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:58.034738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:58.034702 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:59.732123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:12:59.732063 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:01.717380 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:01.717337 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:02.957807 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:02.957771 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:03.128377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:03.128331 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:03.183887 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:13:03.183846 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:13:04.077209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:04.077166 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:04.248596 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:04.248559 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:05.998506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:05.998467 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:07.548241 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:07.548145 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:09.313643 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:09.313593 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:10.887728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:10.887691 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:12.374035 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:12.373989 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:12.534901 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:12.534850 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:13.069259 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:13.069223 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:13.476716 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:13.476624 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:13:13.477110 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:13.476723 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:13.608420 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:13.608382 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:13.749517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:13.749355 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:14.241145 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:14.241101 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:15.899193 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:15.898690 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:17.433661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:17.433626 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:17.496983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:17.496944 5978 generic.go:356] "Generic (PLEG): container finished" podID="4994c0aa-deec-4f24-8422-e5be874c350c" containerID="c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf" exitCode=0 Oct 31 01:13:17.497174 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:17.497016 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" event={"ID":"4994c0aa-deec-4f24-8422-e5be874c350c","Type":"ContainerDied","Data":"c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf"} Oct 31 01:13:17.497520 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:17.497502 5978 scope.go:122] "RemoveContainer" containerID="c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf" Oct 31 01:13:17.499441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:17.499406 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:13:17.499583 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:17.499534 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:13:17.499583 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:17.499575 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:13:17.626474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:17.623860 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:18.085433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.085392 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:18.197811 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.197778 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c"] Oct 31 01:13:18.202400 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.202374 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.221503 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.221468 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-714f8f-openshift-default-dockercfg-687wt\"" Oct 31 01:13:18.226861 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.226821 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c"] Oct 31 01:13:18.341559 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.321126 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-podinfo\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.342017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.341980 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-data\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.342392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.342369 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-token\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.342647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.342624 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-credential-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.342815 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.342798 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-certs\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.342985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.342970 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istiod-ca-cert\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.343107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.343092 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.343249 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.343236 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-envoy\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.344615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.343438 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfrs\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-kube-api-access-mpfrs\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.433377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.433342 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr"] Oct 31 01:13:18.433893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.433856 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" containerID="cri-o://f9683bb7900f277e80a81a2293d27e40c17718831f329dc409440c514b7510c4" gracePeriod=30 Oct 31 01:13:18.446127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.446079 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-podinfo\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.446434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.446408 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-data\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.446872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.446852 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-token\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.449017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.448995 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-credential-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.449197 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.449182 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-certs\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.451554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.451529 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istiod-ca-cert\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.451739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.451721 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.451883 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.451870 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-envoy\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.453331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.453309 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfrs\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-kube-api-access-mpfrs\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.453919 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.447394 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-data\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.454075 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.451178 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-credential-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.454189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.452855 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istiod-ca-cert\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.454287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.449724 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-certs\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.454380 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.452235 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-socket\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.472419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.472372 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-envoy\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.472824 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.472776 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-token\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.473727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.473696 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-podinfo\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.482471 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.482427 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfrs\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-kube-api-access-mpfrs\") pod \"e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.507580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.507549 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" event={"ID":"4994c0aa-deec-4f24-8422-e5be874c350c","Type":"ContainerStarted","Data":"a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186"} Oct 31 01:13:18.517285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.517247 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:18.651023 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.650937 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:13:18.651179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.651145 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:13:18.651227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.651197 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 01:13:18.938252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.938223 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c"] Oct 31 01:13:18.942217 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:13:18.942185 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430b24ac_3a49_4be1_badd_5bcebb7c5ef9.slice/crio-d610086524f42bc14c1e37c8b99b26fac5402b864b6c2d4de3fecac055344cf8 WatchSource:0}: Error finding container d610086524f42bc14c1e37c8b99b26fac5402b864b6c2d4de3fecac055344cf8: Status 404 returned error can't find the container with id d610086524f42bc14c1e37c8b99b26fac5402b864b6c2d4de3fecac055344cf8 Oct 31 01:13:18.945841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.945798 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:13:18.945952 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.945880 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:13:18.945952 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:18.945913 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:13:19.447691 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:19.446517 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:19.541482 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:19.527517 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" event={"ID":"430b24ac-3a49-4be1-badd-5bcebb7c5ef9","Type":"ContainerStarted","Data":"ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2"} Oct 31 01:13:19.541482 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:19.527580 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" event={"ID":"430b24ac-3a49-4be1-badd-5bcebb7c5ef9","Type":"ContainerStarted","Data":"d610086524f42bc14c1e37c8b99b26fac5402b864b6c2d4de3fecac055344cf8"} Oct 31 01:13:19.651526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:19.651487 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:13:19.651713 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:19.651546 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 01:13:20.047477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:20.045325 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:20.198240 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:20.198199 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:20.530478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:20.526375 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:20.531058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:20.530614 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 01:13:20.531058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:20.530674 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 01:13:20.666785 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:20.661961 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:13:20.666785 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:20.662026 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 01:13:20.881820 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:20.881783 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62hfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:13:20.882012 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:20.881855 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:13:21.238027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.237888 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" podStartSLOduration=3.237865885 podStartE2EDuration="3.237865885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:19.579813365 +0000 UTC m=+3217.277927131" watchObservedRunningTime="2026-10-31 01:13:21.237865885 +0000 UTC m=+3218.935979652" Oct 31 01:13:21.258849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.258809 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl"] Oct 31 01:13:21.270480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.267698 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.285273 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.285193 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl"] Oct 31 01:13:21.285613 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.285587 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-498580-openshift-default-dockercfg-8p4s8\"" Oct 31 01:13:21.302048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.302016 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-certs\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.302307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.302282 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-envoy\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.302463 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.302324 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xfn\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-kube-api-access-j4xfn\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.302463 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.302348 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-podinfo\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.302463 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.302372 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-token\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.302664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.302472 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-data\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.302664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.302499 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.302664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.302519 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-credential-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.302664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.302563 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istiod-ca-cert\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.403728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.403693 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istiod-ca-cert\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.403949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.403877 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-certs\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.403956 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-envoy\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404006 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xfn\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-kube-api-access-j4xfn\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404043 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-podinfo\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404083 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-token\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404189 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-data\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404203 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-certs\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404234 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404413 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404266 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-credential-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404491 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404448 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istiod-ca-cert\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404603 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-data\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404754 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.404847 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.404758 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-credential-socket\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.406554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.406532 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-envoy\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.406689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.406669 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-podinfo\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.470052 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.470010 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xfn\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-kube-api-access-j4xfn\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.471052 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.471026 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-token\") pod \"e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.535338 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.534818 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 01:13:21.535338 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.534886 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 01:13:21.563138 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.563099 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:21.597478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.596783 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:21.665506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.653174 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:13:21.665506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:21.653235 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 01:13:22.072251 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.071994 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl"] Oct 31 01:13:22.073056 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:13:22.073029 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5879ea_e13f_4fd7_ad1e_8c5a0f4fc8c6.slice/crio-16ca5fecdf8a94678b6c9579936b6451eb25f0cdea468f19fd4a0e496e97fa22 WatchSource:0}: Error finding container 16ca5fecdf8a94678b6c9579936b6451eb25f0cdea468f19fd4a0e496e97fa22: Status 404 returned error can't find the container with id 16ca5fecdf8a94678b6c9579936b6451eb25f0cdea468f19fd4a0e496e97fa22 Oct 31 01:13:22.076967 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.076932 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:13:22.077130 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.077112 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:13:22.077231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.077219 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:13:22.519472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.518930 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 01:13:22.519472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.519000 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 01:13:22.608982 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.608925 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" event={"ID":"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6","Type":"ContainerStarted","Data":"46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677"} Oct 31 01:13:22.609543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.609006 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" event={"ID":"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6","Type":"ContainerStarted","Data":"16ca5fecdf8a94678b6c9579936b6451eb25f0cdea468f19fd4a0e496e97fa22"} Oct 31 01:13:22.651477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.651050 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:13:22.651477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.651126 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 01:13:22.659487 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:22.657231 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" podStartSLOduration=1.657200191 podStartE2EDuration="1.657200191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:22.655254566 +0000 UTC m=+3220.353368325" watchObservedRunningTime="2026-10-31 01:13:22.657200191 +0000 UTC m=+3220.355313957" Oct 31 01:13:23.093509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.089290 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:23.519473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.518962 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 01:13:23.519473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.519034 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 01:13:23.597821 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.597783 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:23.599340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.599318 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:13:23.599417 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.599362 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:13:23.652478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.651931 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 01:13:23.652478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.651997 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 01:13:23.689477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.689407 5978 generic.go:356] "Generic (PLEG): container finished" podID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerID="f9683bb7900f277e80a81a2293d27e40c17718831f329dc409440c514b7510c4" exitCode=0 Oct 31 01:13:23.690129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:23.690097 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" event={"ID":"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc","Type":"ContainerDied","Data":"f9683bb7900f277e80a81a2293d27e40c17718831f329dc409440c514b7510c4"} Oct 31 01:13:24.385478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.385423 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 01:13:24.490494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.487701 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-podinfo\") pod \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " Oct 31 01:13:24.490494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.487856 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-credential-socket\") pod \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " Oct 31 01:13:24.490494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.487926 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-envoy\") pod \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " Oct 31 01:13:24.490494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.487964 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-data\") pod \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " Oct 31 01:13:24.490494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.488026 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-certs\") pod \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " Oct 31 01:13:24.490494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.488160 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-token\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-token\") pod \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " Oct 31 01:13:24.490494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.488240 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-kube-api-access-4m2gj\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-kube-api-access-4m2gj\") pod \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " Oct 31 01:13:24.490494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.488269 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-socket\") pod \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " Oct 31 01:13:24.490494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.488308 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istiod-ca-cert\") pod \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\" (UID: \"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc\") " Oct 31 01:13:24.491113 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.491016 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-credential-socket" pod "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" (UID: "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:24.531478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.509563 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-data" pod "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" (UID: "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:24.563045 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.562950 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istiod-ca-cert" pod "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" (UID: "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:24.566120 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.564576 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-socket" pod "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" (UID: "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:24.566120 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.565758 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-certs" pod "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" (UID: "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:24.572194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.572133 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-envoy" pod "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" (UID: "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:13:24.583507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.583273 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-token" pod "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" (UID: "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:24.583507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.583402 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-kube-api-access-4m2gj" pod "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" (UID: "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc"). InnerVolumeSpecName "kube-api-access-4m2gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:24.583507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.583505 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 01:13:24.583814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.583558 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 01:13:24.589392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.589272 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-podinfo" pod "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" (UID: "6686fe8f-e8e9-4f29-8760-efd0d88ac1cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:13:24.590700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.590621 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:13:24.590700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.590647 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m2gj\" (UniqueName: \"kubernetes.io/projected/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-kube-api-access-4m2gj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:13:24.590700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.590664 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:13:24.590700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.590676 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:13:24.590700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.590691 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:13:24.591015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.590706 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:13:24.591015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.590719 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:13:24.591015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.590734 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:13:24.591015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.590749 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:13:24.598493 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.597866 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:13:24.598493 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.597928 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:13:24.667791 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.667750 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:13:24.668611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.668585 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:13:24.669661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.669630 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:13:24.697720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.697694 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" Oct 31 01:13:24.697891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.697734 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr" event={"ID":"6686fe8f-e8e9-4f29-8760-efd0d88ac1cc","Type":"ContainerDied","Data":"9db98b73a3ba2771cd763397bc177423851d84e22e81283bef1ea39a4e6881a2"} Oct 31 01:13:24.697891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.697794 5978 scope.go:122] "RemoveContainer" containerID="f9683bb7900f277e80a81a2293d27e40c17718831f329dc409440c514b7510c4" Oct 31 01:13:24.845478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.843515 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr"] Oct 31 01:13:24.876333 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.876278 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62hfr"] Oct 31 01:13:24.878808 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:24.878599 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:25.519488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:25.518886 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 01:13:25.519488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:25.518947 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 01:13:25.597852 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:25.597812 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:13:25.598030 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:25.597868 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:13:26.449924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:26.449891 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" path="/var/lib/kubelet/pods/6686fe8f-e8e9-4f29-8760-efd0d88ac1cc/volumes" Oct 31 01:13:26.498709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:26.498674 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:26.550475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:26.549713 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:26.565943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:26.551158 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:26.565943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:26.552299 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:13:26.597578 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:26.597546 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:13:26.597818 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:26.597602 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:13:27.598425 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:27.598387 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:13:27.598870 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:27.598445 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:13:28.036535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:28.035986 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:28.597703 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:28.597665 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:13:28.597898 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:28.597729 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:13:29.576500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:29.566669 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:29.604539 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:29.604509 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:29.604991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:29.604965 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:29.605667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:29.605645 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:13:31.202382 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:31.202343 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:32.760234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:32.760193 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:32.940217 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:32.940172 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:33.212860 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:13:33.212817 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:13:34.309166 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:34.309121 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:35.903969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:35.903926 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:37.525638 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:37.525595 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:39.118886 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:39.115797 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:41.060778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:41.060733 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:43.196006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:43.195956 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:44.940135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:44.940044 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:46.725365 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:46.725322 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:48.334898 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:48.334857 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:48.911126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:48.907606 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:49.400833 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:49.400795 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:49.549552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:49.549510 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:49.813640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:49.813548 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:50.018069 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:50.018024 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:50.535873 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:50.535834 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:50.773490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:50.768860 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:51.891543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:51.891493 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:53.842529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:53.842207 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:54.837985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:54.837950 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:55.007605 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:55.007559 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:55.757273 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:55.757235 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:57.246355 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:57.246319 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:57.555051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:57.554967 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:57.717036 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:57.716977 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:57.897290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:57.897253 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:59.394080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:13:59.394033 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:00.652132 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:00.652095 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt"] Oct 31 01:14:00.652682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:00.652434 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" podUID="c009cbcf-83a3-4d7d-bcb0-205189531fdb" containerName="maas-api" containerID="cri-o://ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561" gracePeriod=30 Oct 31 01:14:00.765397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:00.765359 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:00.838413 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:00.838380 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st"] Oct 31 01:14:00.838678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:00.838649 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" podUID="1bf12e68-26b5-4104-82f2-a7f7d41fdb1e" containerName="payload-pre-processing" containerID="cri-o://3faaa4e073313ae8c9441fd4cb9dbfc36f2a70d6c2f77b6c8d26a35542b8e752" gracePeriod=30 Oct 31 01:14:00.952240 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:00.952202 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:01.061004 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.060970 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:14:01.103538 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.103502 5978 generic.go:356] "Generic (PLEG): container finished" podID="c009cbcf-83a3-4d7d-bcb0-205189531fdb" containerID="ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561" exitCode=0 Oct 31 01:14:01.103825 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.103554 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" Oct 31 01:14:01.103825 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.103681 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" event={"ID":"c009cbcf-83a3-4d7d-bcb0-205189531fdb","Type":"ContainerDied","Data":"ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561"} Oct 31 01:14:01.103825 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.103815 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt" event={"ID":"c009cbcf-83a3-4d7d-bcb0-205189531fdb","Type":"ContainerDied","Data":"66434dd48014af09ee8d008a33a7b68ead8398be7c99d9f7a83fa6ecde8032fa"} Oct 31 01:14:01.104039 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.103868 5978 scope.go:122] "RemoveContainer" containerID="ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561" Oct 31 01:14:01.147878 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.145000 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:01.173910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.161339 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c009cbcf-83a3-4d7d-bcb0-205189531fdb-kube-api-access-6ws72\" (UniqueName: \"kubernetes.io/projected/c009cbcf-83a3-4d7d-bcb0-205189531fdb-kube-api-access-6ws72\") pod \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " Oct 31 01:14:01.173910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.161539 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls\") pod \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " Oct 31 01:14:01.173910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.161800 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-metrics-certs\") pod \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " Oct 31 01:14:01.173910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.161836 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c009cbcf-83a3-4d7d-bcb0-205189531fdb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c009cbcf-83a3-4d7d-bcb0-205189531fdb-openshift-service-ca\") pod \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\" (UID: \"c009cbcf-83a3-4d7d-bcb0-205189531fdb\") " Oct 31 01:14:01.173910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.163894 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c009cbcf-83a3-4d7d-bcb0-205189531fdb-openshift-service-ca" pod "c009cbcf-83a3-4d7d-bcb0-205189531fdb" (UID: "c009cbcf-83a3-4d7d-bcb0-205189531fdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:01.173910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.166259 5978 generic.go:356] "Generic (PLEG): container finished" podID="1bf12e68-26b5-4104-82f2-a7f7d41fdb1e" containerID="3faaa4e073313ae8c9441fd4cb9dbfc36f2a70d6c2f77b6c8d26a35542b8e752" exitCode=0 Oct 31 01:14:01.173910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.166493 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" event={"ID":"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e","Type":"ContainerDied","Data":"3faaa4e073313ae8c9441fd4cb9dbfc36f2a70d6c2f77b6c8d26a35542b8e752"} Oct 31 01:14:01.174660 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.174569 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-metrics-certs" pod "c009cbcf-83a3-4d7d-bcb0-205189531fdb" (UID: "c009cbcf-83a3-4d7d-bcb0-205189531fdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:01.260704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.260560 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls" pod "c009cbcf-83a3-4d7d-bcb0-205189531fdb" (UID: "c009cbcf-83a3-4d7d-bcb0-205189531fdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:01.267642 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.267027 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c009cbcf-83a3-4d7d-bcb0-205189531fdb-kube-api-access-6ws72" pod "c009cbcf-83a3-4d7d-bcb0-205189531fdb" (UID: "c009cbcf-83a3-4d7d-bcb0-205189531fdb"). InnerVolumeSpecName "kube-api-access-6ws72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:01.269175 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.268968 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:14:01.269175 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.269001 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c009cbcf-83a3-4d7d-bcb0-205189531fdb-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:14:01.269175 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.269021 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ws72\" (UniqueName: \"kubernetes.io/projected/c009cbcf-83a3-4d7d-bcb0-205189531fdb-kube-api-access-6ws72\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:14:01.269175 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.269037 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c009cbcf-83a3-4d7d-bcb0-205189531fdb-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:14:01.277834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.277488 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:14:01.284239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.283992 5978 scope.go:122] "RemoveContainer" containerID="ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561" Oct 31 01:14:01.284577 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:14:01.284447 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561\": container with ID starting with ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561 not found: ID does not exist" containerID="ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561" Oct 31 01:14:01.284577 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.284519 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561"} err="failed to get container status \"ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561\": rpc error: code = NotFound desc = could not find container \"ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561\": container with ID starting with ab51a02664950bdf23c6a1021cdb4a415ae1469b5ade7afa74cc43d5128db561 not found: ID does not exist" Oct 31 01:14:01.335539 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.334324 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:01.370556 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.370505 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-kube-api-access-5bkfq\" (UniqueName: \"kubernetes.io/projected/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-kube-api-access-5bkfq\") pod \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\" (UID: \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\") " Oct 31 01:14:01.370755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.370634 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-plugins-config-volume\") pod \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\" (UID: \"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e\") " Oct 31 01:14:01.371004 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.370981 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-plugins-config-volume" pod "1bf12e68-26b5-4104-82f2-a7f7d41fdb1e" (UID: "1bf12e68-26b5-4104-82f2-a7f7d41fdb1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:01.371428 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.371406 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:14:01.372640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.372616 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-kube-api-access-5bkfq" pod "1bf12e68-26b5-4104-82f2-a7f7d41fdb1e" (UID: "1bf12e68-26b5-4104-82f2-a7f7d41fdb1e"). InnerVolumeSpecName "kube-api-access-5bkfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:01.447532 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.447495 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt"] Oct 31 01:14:01.452383 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.452349 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f915-6c664865-h8lwt"] Oct 31 01:14:01.473561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.473527 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bkfq\" (UniqueName: \"kubernetes.io/projected/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e-kube-api-access-5bkfq\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:14:01.474725 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.474693 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:01.626668 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:01.626628 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:02.012780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:02.012674 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:02.178300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:02.178260 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" Oct 31 01:14:02.178300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:02.178284 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st" event={"ID":"1bf12e68-26b5-4104-82f2-a7f7d41fdb1e","Type":"ContainerDied","Data":"dc2cbb70e38b681fe50a034839f23a060d0e723456c4c3c506cd33131d097505"} Oct 31 01:14:02.178648 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:02.178349 5978 scope.go:122] "RemoveContainer" containerID="3faaa4e073313ae8c9441fd4cb9dbfc36f2a70d6c2f77b6c8d26a35542b8e752" Oct 31 01:14:02.226902 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:02.226868 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st"] Oct 31 01:14:02.235930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:02.235904 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f915-555585ff79-9k9st"] Oct 31 01:14:02.428987 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:02.428952 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf12e68-26b5-4104-82f2-a7f7d41fdb1e" path="/var/lib/kubelet/pods/1bf12e68-26b5-4104-82f2-a7f7d41fdb1e/volumes" Oct 31 01:14:02.429437 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:02.429423 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c009cbcf-83a3-4d7d-bcb0-205189531fdb" path="/var/lib/kubelet/pods/c009cbcf-83a3-4d7d-bcb0-205189531fdb/volumes" Oct 31 01:14:03.189731 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:14:03.189696 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:14:03.284184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:03.284150 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:05.080740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:05.080696 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:05.418268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:05.418163 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:06.694024 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:06.693982 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:08.325030 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:08.324492 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:08.841529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:08.841492 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:08.996107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:08.996077 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:09.237659 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:09.237557 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:10.010498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:10.010446 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:11.682274 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:11.682238 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:13.328239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:13.323251 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:13.711534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:13.711433 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:15.017168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:15.017128 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:16.720805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:16.720772 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:18.389251 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:18.389213 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:21.862562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:21.862516 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:22.015142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:22.015109 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:23.914076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:23.914035 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:25.081309 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:25.081276 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:25.265872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:25.265754 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:25.958407 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:25.958367 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:26.226578 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.226483 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:26.804509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.802447 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:26.983687 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.983599 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt"] Oct 31 01:14:26.985609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.985584 5978 cpu_manager.go:414] "containerMap: removing container" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" Oct 31 01:14:26.985609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.985607 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" Oct 31 01:14:26.985773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.985625 5978 cpu_manager.go:414] "containerMap: removing container" podUID="1bf12e68-26b5-4104-82f2-a7f7d41fdb1e" containerName="payload-pre-processing" Oct 31 01:14:26.985773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.985633 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf12e68-26b5-4104-82f2-a7f7d41fdb1e" containerName="payload-pre-processing" Oct 31 01:14:26.985773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.985655 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c009cbcf-83a3-4d7d-bcb0-205189531fdb" containerName="maas-api" Oct 31 01:14:26.985773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.985659 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c009cbcf-83a3-4d7d-bcb0-205189531fdb" containerName="maas-api" Oct 31 01:14:26.985914 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.985800 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="6686fe8f-e8e9-4f29-8760-efd0d88ac1cc" containerName="istio-proxy" Oct 31 01:14:26.985914 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.985809 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf12e68-26b5-4104-82f2-a7f7d41fdb1e" containerName="payload-pre-processing" Oct 31 01:14:26.985914 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.985821 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c009cbcf-83a3-4d7d-bcb0-205189531fdb" containerName="maas-api" Oct 31 01:14:26.988978 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:26.988955 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.004000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.003968 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-3c90d8-openshift-default-dockercfg-vpjjh\"" Oct 31 01:14:27.021224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.021189 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt"] Oct 31 01:14:27.112471 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.112406 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.112690 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.112563 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-data\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.112690 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.112662 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istiod-ca-cert\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.112841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.112724 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5btn\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-kube-api-access-z5btn\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.112909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.112886 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-podinfo\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.112973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.112917 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-envoy\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.112973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.112955 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-token\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.113094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.113022 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-certs\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.113157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.113130 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-credential-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.218483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.214778 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-credential-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.218483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.214847 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.218483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.214876 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-data\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.218483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.214967 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istiod-ca-cert\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.218483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.215025 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5btn\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-kube-api-access-z5btn\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.218483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.215209 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-podinfo\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.218483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.215240 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-envoy\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.218483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.215283 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-token\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.218483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.215344 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-certs\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.222549 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.222503 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istiod-ca-cert\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.225373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.225031 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-certs\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.225373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.225285 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-credential-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.225631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.225537 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-socket\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.226049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.225842 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-data\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.250510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.248061 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-podinfo\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.287475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.278161 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5btn\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-kube-api-access-z5btn\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.287475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.278803 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-envoy\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.316919 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.315547 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-token\") pod \"e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.443361 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.443279 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:27.603687 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.603636 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:27.774337 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.774037 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:27.937726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.937694 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt"] Oct 31 01:14:27.940347 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:14:27.940315 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9cb0b8_c32f_4e4c_bd46_91239c5b45ab.slice/crio-be8be46f5b98c6aaeb5c95c7fe008fd6d4fd3d1ccbf2ed57b59bebb2f333ab37 WatchSource:0}: Error finding container be8be46f5b98c6aaeb5c95c7fe008fd6d4fd3d1ccbf2ed57b59bebb2f333ab37: Status 404 returned error can't find the container with id be8be46f5b98c6aaeb5c95c7fe008fd6d4fd3d1ccbf2ed57b59bebb2f333ab37 Oct 31 01:14:27.943328 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.943291 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:14:27.943481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.943386 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:14:27.943569 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:27.943495 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:14:28.381306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:28.381269 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:28.440548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:28.440514 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" event={"ID":"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab","Type":"ContainerStarted","Data":"aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c"} Oct 31 01:14:28.441257 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:28.440565 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" event={"ID":"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab","Type":"ContainerStarted","Data":"be8be46f5b98c6aaeb5c95c7fe008fd6d4fd3d1ccbf2ed57b59bebb2f333ab37"} Oct 31 01:14:28.484990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:28.484922 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" podStartSLOduration=2.4849079339999998 podStartE2EDuration="2.484907934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:28.480655515 +0000 UTC m=+3286.178769326" watchObservedRunningTime="2026-10-31 01:14:28.484907934 +0000 UTC m=+3286.183021698" Oct 31 01:14:28.605656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:28.605621 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:29.606010 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:29.605967 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:14:29.606553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:29.606030 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 01:14:30.139948 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:30.139912 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:30.604342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:30.604305 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:14:30.604603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:30.604378 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 01:14:31.236184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:31.236087 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:31.395441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:31.395402 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:31.604859 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:31.604822 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:14:31.605057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:31.604885 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 01:14:31.922531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:31.922177 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:32.604781 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:32.604745 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:14:32.605173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:32.604811 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 01:14:32.937409 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:32.937294 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:33.085271 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.085236 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:33.269530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.268660 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz"] Oct 31 01:14:33.332183 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.332152 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:33.360534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.360504 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fa97ea\"" Oct 31 01:14:33.360773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.360707 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fa97ea-dockercfg-xqpzr\"" Oct 31 01:14:33.363771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.363749 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz"] Oct 31 01:14:33.400447 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:14:33.400413 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:14:33.463572 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.463542 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmnh\" (UniqueName: \"kubernetes.io/projected/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-kube-api-access-dpmnh\") pod \"payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz\" (UID: \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:33.463760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.463585 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz\" (UID: \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:33.565077 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.564981 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmnh\" (UniqueName: \"kubernetes.io/projected/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-kube-api-access-dpmnh\") pod \"payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz\" (UID: \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:33.565077 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.565040 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz\" (UID: \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:33.565820 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.565798 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz\" (UID: \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:33.590202 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.590172 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmnh\" (UniqueName: \"kubernetes.io/projected/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-kube-api-access-dpmnh\") pod \"payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz\" (UID: \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:33.604330 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.604291 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 01:14:33.604507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.604374 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 01:14:33.669617 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.669513 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:33.801725 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:33.796111 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:34.001200 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:34.001167 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz"] Oct 31 01:14:34.001558 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:14:34.001532 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4195cf_cae9_4353_8ea4_9fd607e6bf6b.slice/crio-143c35f7c5240ee2164fb3e180e23532a06d9e800f847e111e86531b571087f2 WatchSource:0}: Error finding container 143c35f7c5240ee2164fb3e180e23532a06d9e800f847e111e86531b571087f2: Status 404 returned error can't find the container with id 143c35f7c5240ee2164fb3e180e23532a06d9e800f847e111e86531b571087f2 Oct 31 01:14:34.498473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:34.498030 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" event={"ID":"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b","Type":"ContainerStarted","Data":"b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4"} Oct 31 01:14:34.498473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:34.498135 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" event={"ID":"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b","Type":"ContainerStarted","Data":"143c35f7c5240ee2164fb3e180e23532a06d9e800f847e111e86531b571087f2"} Oct 31 01:14:34.498473 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:34.498423 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:34.538678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:34.538611 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" podStartSLOduration=1.53859243 podStartE2EDuration="1.53859243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:34.53562471 +0000 UTC m=+3292.233738521" watchObservedRunningTime="2026-10-31 01:14:34.53859243 +0000 UTC m=+3292.236706195" Oct 31 01:14:34.680843 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:34.680763 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:34.682875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:34.682403 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:34.686587 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:34.686427 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:14:35.497497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.497465 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq"] Oct 31 01:14:35.501402 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.501379 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.516561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.516536 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-33d44f-serving-cert\"" Oct 31 01:14:35.539937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.530880 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq"] Oct 31 01:14:35.606185 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.606134 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9c8s\" (UniqueName: \"kubernetes.io/projected/c774cdf8-8e12-4b28-87f2-713de41db619-kube-api-access-m9c8s\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.606185 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.606191 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-metrics-certs\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.606610 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.606586 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c774cdf8-8e12-4b28-87f2-713de41db619-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.606747 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.606728 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.622480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.622424 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:35.711434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.711392 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c774cdf8-8e12-4b28-87f2-713de41db619-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.712023 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.711537 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.712023 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.711690 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9c8s\" (UniqueName: \"kubernetes.io/projected/c774cdf8-8e12-4b28-87f2-713de41db619-kube-api-access-m9c8s\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.712023 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.711731 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-metrics-certs\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.713895 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.713478 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c774cdf8-8e12-4b28-87f2-713de41db619-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.714197 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:14:35.714173 5978 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-serving-cert: secret "maas-api-e2e-ipp-b-33d44f-serving-cert" not found Oct 31 01:14:35.714344 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:14:35.714254 5978 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls podName:c774cdf8-8e12-4b28-87f2-713de41db619 nodeName:}" failed. No retries permitted until 2026-10-31 01:14:36.214233616 +0000 UTC m=+3293.912347374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls") pod "maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" (UID: "c774cdf8-8e12-4b28-87f2-713de41db619") : secret "maas-api-e2e-ipp-b-33d44f-serving-cert" not found Oct 31 01:14:35.715630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.715570 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-metrics-certs\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.733675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.733651 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9c8s\" (UniqueName: \"kubernetes.io/projected/c774cdf8-8e12-4b28-87f2-713de41db619-kube-api-access-m9c8s\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:35.774046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.773953 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:35.915874 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:35.915834 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:36.219020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:36.218983 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:36.221393 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:36.221362 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls\") pod \"maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:36.425216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:36.425177 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:36.779740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:36.779715 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq"] Oct 31 01:14:36.781938 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:14:36.781907 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc774cdf8_8e12_4b28_87f2_713de41db619.slice/crio-9ce74997227c8dda72a53407f89e67a23139473e86363e9effd47d9917517712 WatchSource:0}: Error finding container 9ce74997227c8dda72a53407f89e67a23139473e86363e9effd47d9917517712: Status 404 returned error can't find the container with id 9ce74997227c8dda72a53407f89e67a23139473e86363e9effd47d9917517712 Oct 31 01:14:36.783736 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:36.783714 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:14:37.534276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:37.534188 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" event={"ID":"c774cdf8-8e12-4b28-87f2-713de41db619","Type":"ContainerStarted","Data":"82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4"} Oct 31 01:14:37.534276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:37.534240 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" event={"ID":"c774cdf8-8e12-4b28-87f2-713de41db619","Type":"ContainerStarted","Data":"9ce74997227c8dda72a53407f89e67a23139473e86363e9effd47d9917517712"} Oct 31 01:14:37.534562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:37.534291 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:37.616494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:37.610301 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:39.201182 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:39.201141 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:40.918666 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:40.917650 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:42.592860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:42.592812 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:43.547394 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:43.547366 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:14:43.612616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:43.612556 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" podStartSLOduration=8.127875703 podStartE2EDuration="8.612541263s" totalImagesPullingTime="484.66556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:37.592244706 +0000 UTC m=+3295.290358468" watchObservedRunningTime="2026-10-31 01:14:43.612541263 +0000 UTC m=+3301.310655031" Oct 31 01:14:44.520785 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:44.520743 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:44.700354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:44.700313 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:44.845272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:44.845236 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:45.509057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:45.509028 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:14:46.189013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:46.188980 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:47.790849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:47.790786 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:49.381767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:49.381730 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:49.521088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:49.521045 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:50.599125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:50.599099 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:14:50.602370 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:50.602338 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:14:51.100090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:51.100054 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:52.609771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:52.609722 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:54.188852 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:54.188815 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:55.828395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:55.828341 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:57.403864 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:57.403832 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:59.007888 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:14:59.007853 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:00.210849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.210811 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc"] Oct 31 01:15:00.217764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.217733 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d"] Oct 31 01:15:00.217933 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.217873 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:00.222705 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.222676 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:00.234669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.234636 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r"] Oct 31 01:15:00.242037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.242011 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:00.248676 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.248643 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc"] Oct 31 01:15:00.254664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.254635 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d"] Oct 31 01:15:00.287769 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.287739 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4"] Oct 31 01:15:00.292085 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.292060 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:15:00.293924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.293897 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r"] Oct 31 01:15:00.294065 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.294039 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:00.301042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.301012 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d\" (UID: \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:00.301160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.301069 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b52ft\" (UniqueName: \"kubernetes.io/projected/4c897718-f2e1-4b61-b574-0b6a7701e078-kube-api-access-b52ft\") pod \"maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r\" (UID: \"4c897718-f2e1-4b61-b574-0b6a7701e078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:00.301160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.301142 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c897718-f2e1-4b61-b574-0b6a7701e078-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r\" (UID: \"4c897718-f2e1-4b61-b574-0b6a7701e078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:00.301254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.301208 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b968d397-c0cc-48aa-a6fd-591bcb7c551e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc\" (UID: \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:00.301296 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.301266 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7njhs\" (UniqueName: \"kubernetes.io/projected/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-kube-api-access-7njhs\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d\" (UID: \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:00.301296 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.301286 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5xxs\" (UniqueName: \"kubernetes.io/projected/b968d397-c0cc-48aa-a6fd-591bcb7c551e-kube-api-access-h5xxs\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc\" (UID: \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:00.313663 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.313635 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4"] Oct 31 01:15:00.318964 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.318935 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h"] Oct 31 01:15:00.324790 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.324769 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:00.330664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.330639 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw"] Oct 31 01:15:00.341071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.341046 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:00.365017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.364992 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h"] Oct 31 01:15:00.369260 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.369232 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw"] Oct 31 01:15:00.377562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.377533 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj"] Oct 31 01:15:00.388661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.385699 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:00.401902 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.391739 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj"] Oct 31 01:15:00.403426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403392 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8620a009-6b7c-4e05-a7e6-02ce8ea65956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw\" (UID: \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:00.403631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403500 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d\" (UID: \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:00.403631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403544 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52ft\" (UniqueName: \"kubernetes.io/projected/4c897718-f2e1-4b61-b574-0b6a7701e078-kube-api-access-b52ft\") pod \"maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r\" (UID: \"4c897718-f2e1-4b61-b574-0b6a7701e078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:00.403631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403569 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdw2z\" (UniqueName: \"kubernetes.io/projected/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-kube-api-access-gdw2z\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h\" (UID: \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:00.403631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403607 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2bc\" (UniqueName: \"kubernetes.io/projected/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-kube-api-access-gd2bc\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4\" (UID: \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:00.403868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403637 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h\" (UID: \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:00.403868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403695 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c897718-f2e1-4b61-b574-0b6a7701e078-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r\" (UID: \"4c897718-f2e1-4b61-b574-0b6a7701e078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:00.403868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403752 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5kd\" (UniqueName: \"kubernetes.io/projected/8620a009-6b7c-4e05-a7e6-02ce8ea65956-kube-api-access-bz5kd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw\" (UID: \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:00.403868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403792 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b968d397-c0cc-48aa-a6fd-591bcb7c551e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc\" (UID: \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:00.404088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403867 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7njhs\" (UniqueName: \"kubernetes.io/projected/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-kube-api-access-7njhs\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d\" (UID: \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:00.404088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403895 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5xxs\" (UniqueName: \"kubernetes.io/projected/b968d397-c0cc-48aa-a6fd-591bcb7c551e-kube-api-access-h5xxs\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc\" (UID: \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:00.404088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.403925 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4\" (UID: \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:00.405508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.404875 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c897718-f2e1-4b61-b574-0b6a7701e078-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r\" (UID: \"4c897718-f2e1-4b61-b574-0b6a7701e078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:00.405508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.404990 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d\" (UID: \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:00.405724 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.405666 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b968d397-c0cc-48aa-a6fd-591bcb7c551e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc\" (UID: \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:00.426116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.426079 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52ft\" (UniqueName: \"kubernetes.io/projected/4c897718-f2e1-4b61-b574-0b6a7701e078-kube-api-access-b52ft\") pod \"maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r\" (UID: \"4c897718-f2e1-4b61-b574-0b6a7701e078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:00.427306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.427282 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7njhs\" (UniqueName: \"kubernetes.io/projected/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-kube-api-access-7njhs\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d\" (UID: \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:00.427431 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.427372 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5xxs\" (UniqueName: \"kubernetes.io/projected/b968d397-c0cc-48aa-a6fd-591bcb7c551e-kube-api-access-h5xxs\") pod \"maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc\" (UID: \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:00.505777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.505683 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj\" (UID: \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:00.505777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.505759 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdw2z\" (UniqueName: \"kubernetes.io/projected/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-kube-api-access-gdw2z\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h\" (UID: \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:00.505999 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.505823 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2bc\" (UniqueName: \"kubernetes.io/projected/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-kube-api-access-gd2bc\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4\" (UID: \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:00.505999 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.505857 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h\" (UID: \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:00.505999 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.505936 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5kd\" (UniqueName: \"kubernetes.io/projected/8620a009-6b7c-4e05-a7e6-02ce8ea65956-kube-api-access-bz5kd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw\" (UID: \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:00.506255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.506007 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4\" (UID: \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:00.506255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.506193 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqpfn\" (UniqueName: \"kubernetes.io/projected/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-kube-api-access-jqpfn\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj\" (UID: \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:00.506255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.506251 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8620a009-6b7c-4e05-a7e6-02ce8ea65956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw\" (UID: \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:00.506886 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.506860 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4\" (UID: \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:00.507001 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.506884 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8620a009-6b7c-4e05-a7e6-02ce8ea65956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw\" (UID: \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:00.507001 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.506973 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h\" (UID: \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:00.526698 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.526671 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5kd\" (UniqueName: \"kubernetes.io/projected/8620a009-6b7c-4e05-a7e6-02ce8ea65956-kube-api-access-bz5kd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw\" (UID: \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:00.526883 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.526737 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdw2z\" (UniqueName: \"kubernetes.io/projected/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-kube-api-access-gdw2z\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h\" (UID: \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:00.526883 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.526811 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2bc\" (UniqueName: \"kubernetes.io/projected/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-kube-api-access-gd2bc\") pod \"maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4\" (UID: \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:00.544499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.544470 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:00.565691 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.565660 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:00.578579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.578533 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:00.608492 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.608356 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqpfn\" (UniqueName: \"kubernetes.io/projected/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-kube-api-access-jqpfn\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj\" (UID: \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:00.608492 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.608482 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj\" (UID: \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:00.609633 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.609606 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj\" (UID: \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:00.625767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.618146 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:00.634541 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.634508 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqpfn\" (UniqueName: \"kubernetes.io/projected/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-kube-api-access-jqpfn\") pod \"maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj\" (UID: \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:00.647115 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.647080 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:00.666008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.665978 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:00.739146 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.739109 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:00.889940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.889724 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:00.894650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.894614 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc"] Oct 31 01:15:00.924133 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:15:00.924092 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb968d397_c0cc_48aa_a6fd_591bcb7c551e.slice/crio-64e2e816bac7e9f4410153a08cd340673a8c0105b62af84c097d6911a590acae WatchSource:0}: Error finding container 64e2e816bac7e9f4410153a08cd340673a8c0105b62af84c097d6911a590acae: Status 404 returned error can't find the container with id 64e2e816bac7e9f4410153a08cd340673a8c0105b62af84c097d6911a590acae Oct 31 01:15:00.941384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.941353 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d"] Oct 31 01:15:00.950827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.950797 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r"] Oct 31 01:15:00.955711 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:15:00.955544 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c897718_f2e1_4b61_b574_0b6a7701e078.slice/crio-037e2b0d1f7e09f1df904b2e0051e6807fe20c50334ea695e9682b49c2dd4667 WatchSource:0}: Error finding container 037e2b0d1f7e09f1df904b2e0051e6807fe20c50334ea695e9682b49c2dd4667: Status 404 returned error can't find the container with id 037e2b0d1f7e09f1df904b2e0051e6807fe20c50334ea695e9682b49c2dd4667 Oct 31 01:15:00.986681 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:00.986570 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4"] Oct 31 01:15:00.990046 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:15:00.989948 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27563c1_b719_44fa_9cb6_fb1571c7f7d3.slice/crio-82459151ec4c84026e1bf57f615db8281c568f31f6a34ad8b815e50642f262c7 WatchSource:0}: Error finding container 82459151ec4c84026e1bf57f615db8281c568f31f6a34ad8b815e50642f262c7: Status 404 returned error can't find the container with id 82459151ec4c84026e1bf57f615db8281c568f31f6a34ad8b815e50642f262c7 Oct 31 01:15:01.151259 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.151231 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj"] Oct 31 01:15:01.158318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.158295 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h"] Oct 31 01:15:01.170375 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.170299 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw"] Oct 31 01:15:01.214784 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:15:01.214747 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee0a45f_f22a_4f80_afac_4b1a55d8d386.slice/crio-8bc59af517db7ac86539c584faece5f8c20440568abedb20aec788f72fbe09bf WatchSource:0}: Error finding container 8bc59af517db7ac86539c584faece5f8c20440568abedb20aec788f72fbe09bf: Status 404 returned error can't find the container with id 8bc59af517db7ac86539c584faece5f8c20440568abedb20aec788f72fbe09bf Oct 31 01:15:01.216902 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:15:01.216873 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d73a0c_adbb_45c2_8e65_f2e7e59dc2b7.slice/crio-3c6e5d36d478021a652b33c21f4c2f0b70095f8f8391d378de82f76e2cfe9cb5 WatchSource:0}: Error finding container 3c6e5d36d478021a652b33c21f4c2f0b70095f8f8391d378de82f76e2cfe9cb5: Status 404 returned error can't find the container with id 3c6e5d36d478021a652b33c21f4c2f0b70095f8f8391d378de82f76e2cfe9cb5 Oct 31 01:15:01.222954 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:15:01.222924 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8620a009_6b7c_4e05_a7e6_02ce8ea65956.slice/crio-f9ea7e87d648961599cc6ab095dc3732d3c04297248f1f4967933e7b8fdf8fce WatchSource:0}: Error finding container f9ea7e87d648961599cc6ab095dc3732d3c04297248f1f4967933e7b8fdf8fce: Status 404 returned error can't find the container with id f9ea7e87d648961599cc6ab095dc3732d3c04297248f1f4967933e7b8fdf8fce Oct 31 01:15:01.746096 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.746063 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" event={"ID":"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7","Type":"ContainerStarted","Data":"3ba5859f2fc8ab7c1e9e6ab30020d52170028938e3197682c0daecc8051ac163"} Oct 31 01:15:01.746271 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.746121 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" event={"ID":"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7","Type":"ContainerStarted","Data":"3c6e5d36d478021a652b33c21f4c2f0b70095f8f8391d378de82f76e2cfe9cb5"} Oct 31 01:15:01.747790 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.747762 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" event={"ID":"4c897718-f2e1-4b61-b574-0b6a7701e078","Type":"ContainerStarted","Data":"bbde96935d68319cb97a125cdf522a354529dfcc6b5fefbf6ca1d90540b94c4e"} Oct 31 01:15:01.747912 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.747808 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" event={"ID":"4c897718-f2e1-4b61-b574-0b6a7701e078","Type":"ContainerStarted","Data":"037e2b0d1f7e09f1df904b2e0051e6807fe20c50334ea695e9682b49c2dd4667"} Oct 31 01:15:01.749410 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.749382 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" event={"ID":"b968d397-c0cc-48aa-a6fd-591bcb7c551e","Type":"ContainerStarted","Data":"efdf5431a97e573d3ccd916f08a50d1ca2afaba5b5562bddff75b496e39fb8fe"} Oct 31 01:15:01.749529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.749425 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" event={"ID":"b968d397-c0cc-48aa-a6fd-591bcb7c551e","Type":"ContainerStarted","Data":"64e2e816bac7e9f4410153a08cd340673a8c0105b62af84c097d6911a590acae"} Oct 31 01:15:01.750996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.750972 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" event={"ID":"ba5e3387-cb76-4cf6-a796-2b2cfebb634a","Type":"ContainerStarted","Data":"ea98b47062bcc83d15cc9206f5cdf7275847cab2e701bd988a0365ec4c312497"} Oct 31 01:15:01.751094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.751010 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" event={"ID":"ba5e3387-cb76-4cf6-a796-2b2cfebb634a","Type":"ContainerStarted","Data":"53ba9a990f169af80ce306da7066d805e8c179cc6cf8931a17e9c80dc629f1ae"} Oct 31 01:15:01.752620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.752602 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" event={"ID":"8620a009-6b7c-4e05-a7e6-02ce8ea65956","Type":"ContainerStarted","Data":"f85a845003622dca4e731f5a437d7899deb12a6ac284ccc7d06bda9745104319"} Oct 31 01:15:01.752707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.752636 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" event={"ID":"8620a009-6b7c-4e05-a7e6-02ce8ea65956","Type":"ContainerStarted","Data":"f9ea7e87d648961599cc6ab095dc3732d3c04297248f1f4967933e7b8fdf8fce"} Oct 31 01:15:01.753965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.753942 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" event={"ID":"5ee0a45f-f22a-4f80-afac-4b1a55d8d386","Type":"ContainerStarted","Data":"8bc59af517db7ac86539c584faece5f8c20440568abedb20aec788f72fbe09bf"} Oct 31 01:15:01.755481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.755430 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" event={"ID":"c27563c1-b719-44fa-9cb6-fb1571c7f7d3","Type":"ContainerStarted","Data":"b67f0bd120b22c2ecf2fb81ee7e384caeaa2c6b406cf79a3dfba7eda133dd933"} Oct 31 01:15:01.755594 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.755491 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" event={"ID":"c27563c1-b719-44fa-9cb6-fb1571c7f7d3","Type":"ContainerStarted","Data":"82459151ec4c84026e1bf57f615db8281c568f31f6a34ad8b815e50642f262c7"} Oct 31 01:15:01.783036 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.782973 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" podStartSLOduration=1.782957357 podStartE2EDuration="1.782957357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.780690959 +0000 UTC m=+3319.478804738" watchObservedRunningTime="2026-10-31 01:15:01.782957357 +0000 UTC m=+3319.481071122" Oct 31 01:15:01.815803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.815747 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" podStartSLOduration=1.815734358 podStartE2EDuration="1.815734358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.814862238 +0000 UTC m=+3319.512976016" watchObservedRunningTime="2026-10-31 01:15:01.815734358 +0000 UTC m=+3319.513848165" Oct 31 01:15:01.858011 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.857944 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" podStartSLOduration=1.8579242759999999 podStartE2EDuration="1.857924276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.85561681 +0000 UTC m=+3319.553730588" watchObservedRunningTime="2026-10-31 01:15:01.857924276 +0000 UTC m=+3319.556038042" Oct 31 01:15:01.895417 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.895366 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" podStartSLOduration=1.89535068 podStartE2EDuration="1.89535068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.893422762 +0000 UTC m=+3319.591536526" watchObservedRunningTime="2026-10-31 01:15:01.89535068 +0000 UTC m=+3319.593464445" Oct 31 01:15:01.942479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.938442 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" podStartSLOduration=1.938423722 podStartE2EDuration="1.938423722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.935263085 +0000 UTC m=+3319.633376868" watchObservedRunningTime="2026-10-31 01:15:01.938423722 +0000 UTC m=+3319.636537488" Oct 31 01:15:01.971000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:01.970887 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" podStartSLOduration=1.970873451 podStartE2EDuration="1.970873451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.970636973 +0000 UTC m=+3319.668750743" watchObservedRunningTime="2026-10-31 01:15:01.970873451 +0000 UTC m=+3319.668987215" Oct 31 01:15:02.649075 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.627596 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:02.765085 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.765053 5978 generic.go:356] "Generic (PLEG): container finished" podID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerID="ea98b47062bcc83d15cc9206f5cdf7275847cab2e701bd988a0365ec4c312497" exitCode=60 Oct 31 01:15:02.765248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.765139 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" event={"ID":"ba5e3387-cb76-4cf6-a796-2b2cfebb634a","Type":"ContainerDied","Data":"ea98b47062bcc83d15cc9206f5cdf7275847cab2e701bd988a0365ec4c312497"} Oct 31 01:15:02.765515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.765493 5978 scope.go:122] "RemoveContainer" containerID="ea98b47062bcc83d15cc9206f5cdf7275847cab2e701bd988a0365ec4c312497" Oct 31 01:15:02.766931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.766907 5978 generic.go:356] "Generic (PLEG): container finished" podID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerID="f85a845003622dca4e731f5a437d7899deb12a6ac284ccc7d06bda9745104319" exitCode=60 Oct 31 01:15:02.767053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.766984 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" event={"ID":"8620a009-6b7c-4e05-a7e6-02ce8ea65956","Type":"ContainerDied","Data":"f85a845003622dca4e731f5a437d7899deb12a6ac284ccc7d06bda9745104319"} Oct 31 01:15:02.767224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.767208 5978 scope.go:122] "RemoveContainer" containerID="f85a845003622dca4e731f5a437d7899deb12a6ac284ccc7d06bda9745104319" Oct 31 01:15:02.768928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.768909 5978 generic.go:356] "Generic (PLEG): container finished" podID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerID="95835fdda18569ef423c70a38e0cf7194e0bce5d763c3123341303f261627bfa" exitCode=60 Oct 31 01:15:02.769012 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.768997 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" event={"ID":"5ee0a45f-f22a-4f80-afac-4b1a55d8d386","Type":"ContainerDied","Data":"95835fdda18569ef423c70a38e0cf7194e0bce5d763c3123341303f261627bfa"} Oct 31 01:15:02.769285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.769269 5978 scope.go:122] "RemoveContainer" containerID="95835fdda18569ef423c70a38e0cf7194e0bce5d763c3123341303f261627bfa" Oct 31 01:15:02.770979 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.770962 5978 generic.go:356] "Generic (PLEG): container finished" podID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerID="b67f0bd120b22c2ecf2fb81ee7e384caeaa2c6b406cf79a3dfba7eda133dd933" exitCode=60 Oct 31 01:15:02.771057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.771028 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" event={"ID":"c27563c1-b719-44fa-9cb6-fb1571c7f7d3","Type":"ContainerDied","Data":"b67f0bd120b22c2ecf2fb81ee7e384caeaa2c6b406cf79a3dfba7eda133dd933"} Oct 31 01:15:02.771263 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.771248 5978 scope.go:122] "RemoveContainer" containerID="b67f0bd120b22c2ecf2fb81ee7e384caeaa2c6b406cf79a3dfba7eda133dd933" Oct 31 01:15:02.772751 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.772729 5978 generic.go:356] "Generic (PLEG): container finished" podID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerID="3ba5859f2fc8ab7c1e9e6ab30020d52170028938e3197682c0daecc8051ac163" exitCode=60 Oct 31 01:15:02.772865 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.772799 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" event={"ID":"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7","Type":"ContainerDied","Data":"3ba5859f2fc8ab7c1e9e6ab30020d52170028938e3197682c0daecc8051ac163"} Oct 31 01:15:02.773043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.773025 5978 scope.go:122] "RemoveContainer" containerID="3ba5859f2fc8ab7c1e9e6ab30020d52170028938e3197682c0daecc8051ac163" Oct 31 01:15:02.779677 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.779655 5978 generic.go:356] "Generic (PLEG): container finished" podID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerID="bbde96935d68319cb97a125cdf522a354529dfcc6b5fefbf6ca1d90540b94c4e" exitCode=60 Oct 31 01:15:02.779808 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.779740 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" event={"ID":"4c897718-f2e1-4b61-b574-0b6a7701e078","Type":"ContainerDied","Data":"bbde96935d68319cb97a125cdf522a354529dfcc6b5fefbf6ca1d90540b94c4e"} Oct 31 01:15:02.779985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.779972 5978 scope.go:122] "RemoveContainer" containerID="bbde96935d68319cb97a125cdf522a354529dfcc6b5fefbf6ca1d90540b94c4e" Oct 31 01:15:02.781958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.781925 5978 generic.go:356] "Generic (PLEG): container finished" podID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerID="efdf5431a97e573d3ccd916f08a50d1ca2afaba5b5562bddff75b496e39fb8fe" exitCode=60 Oct 31 01:15:02.782060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.781974 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" event={"ID":"b968d397-c0cc-48aa-a6fd-591bcb7c551e","Type":"ContainerDied","Data":"efdf5431a97e573d3ccd916f08a50d1ca2afaba5b5562bddff75b496e39fb8fe"} Oct 31 01:15:02.782281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:02.782265 5978 scope.go:122] "RemoveContainer" containerID="efdf5431a97e573d3ccd916f08a50d1ca2afaba5b5562bddff75b496e39fb8fe" Oct 31 01:15:03.432862 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:03.432830 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:15:03.792093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.792008 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" event={"ID":"8620a009-6b7c-4e05-a7e6-02ce8ea65956","Type":"ContainerStarted","Data":"7c78eeb48d7bc88ab1f6af0b113f7876f7b8ca528a0d165b393b2f879b74ba38"} Oct 31 01:15:03.794195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.794166 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" event={"ID":"5ee0a45f-f22a-4f80-afac-4b1a55d8d386","Type":"ContainerStarted","Data":"20a1c75eb3780036665f4f93028139b7ecb8d276baa90b2fedf8b94e8e6a6a76"} Oct 31 01:15:03.796077 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.796055 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" event={"ID":"c27563c1-b719-44fa-9cb6-fb1571c7f7d3","Type":"ContainerStarted","Data":"368955c666d721934de01864f83df2a51101ce97f0f47bb8375f5d8c88d0607c"} Oct 31 01:15:03.797928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.797908 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" event={"ID":"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7","Type":"ContainerStarted","Data":"e270deb074304d1d703434fab2c4f0ac1483ea17152a74104894fd4a3587537c"} Oct 31 01:15:03.799780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.799759 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" event={"ID":"4c897718-f2e1-4b61-b574-0b6a7701e078","Type":"ContainerStarted","Data":"2bbcd77f9fdf3867c23a555bef67ca0775e25a5f330ab74e51b4c935d22e1c9c"} Oct 31 01:15:03.801566 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.801540 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" event={"ID":"b968d397-c0cc-48aa-a6fd-591bcb7c551e","Type":"ContainerStarted","Data":"7edc18fa98c6ef146f6dba70f57c782b942e32221010522b7e3365d327b8687f"} Oct 31 01:15:03.803332 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.803312 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" event={"ID":"ba5e3387-cb76-4cf6-a796-2b2cfebb634a","Type":"ContainerStarted","Data":"f97b8e59de08aff284e1bca92a829c2d9c73737da52b37c7bd5b81ee7dccfad4"} Oct 31 01:15:03.865541 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.865485 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" podStartSLOduration=3.865471355 podStartE2EDuration="3.865471355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:03.862623309 +0000 UTC m=+3321.560737074" watchObservedRunningTime="2026-10-31 01:15:03.865471355 +0000 UTC m=+3321.563585117" Oct 31 01:15:03.975339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.973632 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8"] Oct 31 01:15:03.975339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:03.973889 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" podUID="8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" containerName="maas-api" containerID="cri-o://175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3" gracePeriod=30 Oct 31 01:15:04.017107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.016885 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:04.184518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.184428 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:04.332549 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.332514 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:04.398091 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.398064 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:15:04.500479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.492987 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-metrics-certs\") pod \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " Oct 31 01:15:04.500479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.493262 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls\") pod \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " Oct 31 01:15:04.500479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.493349 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-openshift-service-ca\") pod \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " Oct 31 01:15:04.500479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.493383 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-kube-api-access-mn2zw\" (UniqueName: \"kubernetes.io/projected/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-kube-api-access-mn2zw\") pod \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\" (UID: \"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00\") " Oct 31 01:15:04.504519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.504406 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-kube-api-access-mn2zw" pod "8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" (UID: "8ec2edd6-67f1-4bd4-b4ab-b64a389ead00"). InnerVolumeSpecName "kube-api-access-mn2zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:04.521616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.521580 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls" pod "8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" (UID: "8ec2edd6-67f1-4bd4-b4ab-b64a389ead00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:04.529475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.524606 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-metrics-certs" pod "8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" (UID: "8ec2edd6-67f1-4bd4-b4ab-b64a389ead00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:04.544477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.533671 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-openshift-service-ca" pod "8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" (UID: "8ec2edd6-67f1-4bd4-b4ab-b64a389ead00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:04.607104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.606941 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:04.607104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.606983 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:04.607104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.607002 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn2zw\" (UniqueName: \"kubernetes.io/projected/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-kube-api-access-mn2zw\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:04.607104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.607016 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:04.778672 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.778623 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:04.817398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.817362 5978 generic.go:356] "Generic (PLEG): container finished" podID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerID="7c78eeb48d7bc88ab1f6af0b113f7876f7b8ca528a0d165b393b2f879b74ba38" exitCode=60 Oct 31 01:15:04.817986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.817441 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" event={"ID":"8620a009-6b7c-4e05-a7e6-02ce8ea65956","Type":"ContainerDied","Data":"7c78eeb48d7bc88ab1f6af0b113f7876f7b8ca528a0d165b393b2f879b74ba38"} Oct 31 01:15:04.817986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.817524 5978 scope.go:122] "RemoveContainer" containerID="f85a845003622dca4e731f5a437d7899deb12a6ac284ccc7d06bda9745104319" Oct 31 01:15:04.817986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.817827 5978 scope.go:122] "RemoveContainer" containerID="7c78eeb48d7bc88ab1f6af0b113f7876f7b8ca528a0d165b393b2f879b74ba38" Oct 31 01:15:04.820213 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:04.820183 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw_odh-ai-gateway-infra(8620a009-6b7c-4e05-a7e6-02ce8ea65956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" Oct 31 01:15:04.820567 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.820540 5978 generic.go:356] "Generic (PLEG): container finished" podID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerID="20a1c75eb3780036665f4f93028139b7ecb8d276baa90b2fedf8b94e8e6a6a76" exitCode=60 Oct 31 01:15:04.821536 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.821329 5978 scope.go:122] "RemoveContainer" containerID="20a1c75eb3780036665f4f93028139b7ecb8d276baa90b2fedf8b94e8e6a6a76" Oct 31 01:15:04.821794 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:04.821767 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj_odh-ai-gateway-infra(5ee0a45f-f22a-4f80-afac-4b1a55d8d386)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" Oct 31 01:15:04.821915 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.821821 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" event={"ID":"5ee0a45f-f22a-4f80-afac-4b1a55d8d386","Type":"ContainerDied","Data":"20a1c75eb3780036665f4f93028139b7ecb8d276baa90b2fedf8b94e8e6a6a76"} Oct 31 01:15:04.824718 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.824693 5978 generic.go:356] "Generic (PLEG): container finished" podID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerID="368955c666d721934de01864f83df2a51101ce97f0f47bb8375f5d8c88d0607c" exitCode=60 Oct 31 01:15:04.824827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.824762 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" event={"ID":"c27563c1-b719-44fa-9cb6-fb1571c7f7d3","Type":"ContainerDied","Data":"368955c666d721934de01864f83df2a51101ce97f0f47bb8375f5d8c88d0607c"} Oct 31 01:15:04.825110 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.825090 5978 scope.go:122] "RemoveContainer" containerID="368955c666d721934de01864f83df2a51101ce97f0f47bb8375f5d8c88d0607c" Oct 31 01:15:04.825518 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:04.825491 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4_odh-ai-gateway-infra(c27563c1-b719-44fa-9cb6-fb1571c7f7d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" Oct 31 01:15:04.826973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.826953 5978 generic.go:356] "Generic (PLEG): container finished" podID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerID="e270deb074304d1d703434fab2c4f0ac1483ea17152a74104894fd4a3587537c" exitCode=60 Oct 31 01:15:04.827049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.827034 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" event={"ID":"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7","Type":"ContainerDied","Data":"e270deb074304d1d703434fab2c4f0ac1483ea17152a74104894fd4a3587537c"} Oct 31 01:15:04.827319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.827301 5978 scope.go:122] "RemoveContainer" containerID="e270deb074304d1d703434fab2c4f0ac1483ea17152a74104894fd4a3587537c" Oct 31 01:15:04.827682 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:04.827660 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h_odh-ai-gateway-infra(f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" Oct 31 01:15:04.829167 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.829150 5978 generic.go:356] "Generic (PLEG): container finished" podID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerID="2bbcd77f9fdf3867c23a555bef67ca0775e25a5f330ab74e51b4c935d22e1c9c" exitCode=60 Oct 31 01:15:04.829260 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.829208 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" event={"ID":"4c897718-f2e1-4b61-b574-0b6a7701e078","Type":"ContainerDied","Data":"2bbcd77f9fdf3867c23a555bef67ca0775e25a5f330ab74e51b4c935d22e1c9c"} Oct 31 01:15:04.829422 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.829406 5978 scope.go:122] "RemoveContainer" containerID="2bbcd77f9fdf3867c23a555bef67ca0775e25a5f330ab74e51b4c935d22e1c9c" Oct 31 01:15:04.829703 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:04.829681 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r_odh-ai-gateway-infra(4c897718-f2e1-4b61-b574-0b6a7701e078)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" Oct 31 01:15:04.830818 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.830800 5978 generic.go:356] "Generic (PLEG): container finished" podID="8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" containerID="175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3" exitCode=0 Oct 31 01:15:04.830895 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.830851 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" Oct 31 01:15:04.830895 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.830880 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" event={"ID":"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00","Type":"ContainerDied","Data":"175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3"} Oct 31 01:15:04.831034 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.830928 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8" event={"ID":"8ec2edd6-67f1-4bd4-b4ab-b64a389ead00","Type":"ContainerDied","Data":"a89d9cc17e1c1eefb497d406d1ed7ca688626d90ffca56f60f04ccbc1d1fae41"} Oct 31 01:15:04.833767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.832918 5978 generic.go:356] "Generic (PLEG): container finished" podID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerID="7edc18fa98c6ef146f6dba70f57c782b942e32221010522b7e3365d327b8687f" exitCode=60 Oct 31 01:15:04.833767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.832997 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" event={"ID":"b968d397-c0cc-48aa-a6fd-591bcb7c551e","Type":"ContainerDied","Data":"7edc18fa98c6ef146f6dba70f57c782b942e32221010522b7e3365d327b8687f"} Oct 31 01:15:04.833767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.833186 5978 scope.go:122] "RemoveContainer" containerID="7edc18fa98c6ef146f6dba70f57c782b942e32221010522b7e3365d327b8687f" Oct 31 01:15:04.833767 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:04.833384 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc_odh-ai-gateway-infra(b968d397-c0cc-48aa-a6fd-591bcb7c551e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" Oct 31 01:15:04.835216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.835177 5978 generic.go:356] "Generic (PLEG): container finished" podID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerID="f97b8e59de08aff284e1bca92a829c2d9c73737da52b37c7bd5b81ee7dccfad4" exitCode=60 Oct 31 01:15:04.835316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.835212 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" event={"ID":"ba5e3387-cb76-4cf6-a796-2b2cfebb634a","Type":"ContainerDied","Data":"f97b8e59de08aff284e1bca92a829c2d9c73737da52b37c7bd5b81ee7dccfad4"} Oct 31 01:15:04.835531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.835513 5978 scope.go:122] "RemoveContainer" containerID="f97b8e59de08aff284e1bca92a829c2d9c73737da52b37c7bd5b81ee7dccfad4" Oct 31 01:15:04.835793 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:04.835761 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d_odh-ai-gateway-infra(ba5e3387-cb76-4cf6-a796-2b2cfebb634a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" Oct 31 01:15:04.860897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.860874 5978 scope.go:122] "RemoveContainer" containerID="95835fdda18569ef423c70a38e0cf7194e0bce5d763c3123341303f261627bfa" Oct 31 01:15:04.880694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.880668 5978 scope.go:122] "RemoveContainer" containerID="b67f0bd120b22c2ecf2fb81ee7e384caeaa2c6b406cf79a3dfba7eda133dd933" Oct 31 01:15:04.894097 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.894079 5978 scope.go:122] "RemoveContainer" containerID="3ba5859f2fc8ab7c1e9e6ab30020d52170028938e3197682c0daecc8051ac163" Oct 31 01:15:04.933662 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.933625 5978 scope.go:122] "RemoveContainer" containerID="bbde96935d68319cb97a125cdf522a354529dfcc6b5fefbf6ca1d90540b94c4e" Oct 31 01:15:04.952572 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.952543 5978 scope.go:122] "RemoveContainer" containerID="175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3" Oct 31 01:15:04.965838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.965736 5978 scope.go:122] "RemoveContainer" containerID="175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3" Oct 31 01:15:04.966380 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:04.966228 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3\": container with ID starting with 175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3 not found: ID does not exist" containerID="175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3" Oct 31 01:15:04.966380 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.966275 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3"} err="failed to get container status \"175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3\": rpc error: code = NotFound desc = could not find container \"175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3\": container with ID starting with 175d49c0474d7548bb4070729f4e6ad15ea0ac9e79b1fdcdb656ad6923e656c3 not found: ID does not exist" Oct 31 01:15:04.966380 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.966294 5978 scope.go:122] "RemoveContainer" containerID="efdf5431a97e573d3ccd916f08a50d1ca2afaba5b5562bddff75b496e39fb8fe" Oct 31 01:15:04.972463 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.972418 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:04.981685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:04.981540 5978 scope.go:122] "RemoveContainer" containerID="ea98b47062bcc83d15cc9206f5cdf7275847cab2e701bd988a0365ec4c312497" Oct 31 01:15:05.123604 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.123503 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8"] Oct 31 01:15:05.129650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.129615 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75ead8-59895b79bf-hjlv8"] Oct 31 01:15:05.136941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.136901 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:05.479516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.479420 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:05.849363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.849334 5978 scope.go:122] "RemoveContainer" containerID="7edc18fa98c6ef146f6dba70f57c782b942e32221010522b7e3365d327b8687f" Oct 31 01:15:05.849845 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:05.849779 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc_odh-ai-gateway-infra(b968d397-c0cc-48aa-a6fd-591bcb7c551e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" Oct 31 01:15:05.851298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.851275 5978 scope.go:122] "RemoveContainer" containerID="f97b8e59de08aff284e1bca92a829c2d9c73737da52b37c7bd5b81ee7dccfad4" Oct 31 01:15:05.851705 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:05.851678 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d_odh-ai-gateway-infra(ba5e3387-cb76-4cf6-a796-2b2cfebb634a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" Oct 31 01:15:05.853084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.853065 5978 scope.go:122] "RemoveContainer" containerID="7c78eeb48d7bc88ab1f6af0b113f7876f7b8ca528a0d165b393b2f879b74ba38" Oct 31 01:15:05.853361 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:05.853340 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw_odh-ai-gateway-infra(8620a009-6b7c-4e05-a7e6-02ce8ea65956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" Oct 31 01:15:05.854640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.854622 5978 scope.go:122] "RemoveContainer" containerID="20a1c75eb3780036665f4f93028139b7ecb8d276baa90b2fedf8b94e8e6a6a76" Oct 31 01:15:05.854909 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:05.854885 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj_odh-ai-gateway-infra(5ee0a45f-f22a-4f80-afac-4b1a55d8d386)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" Oct 31 01:15:05.856416 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.856395 5978 scope.go:122] "RemoveContainer" containerID="368955c666d721934de01864f83df2a51101ce97f0f47bb8375f5d8c88d0607c" Oct 31 01:15:05.856669 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:05.856647 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4_odh-ai-gateway-infra(c27563c1-b719-44fa-9cb6-fb1571c7f7d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" Oct 31 01:15:05.857967 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.857945 5978 scope.go:122] "RemoveContainer" containerID="e270deb074304d1d703434fab2c4f0ac1483ea17152a74104894fd4a3587537c" Oct 31 01:15:05.858200 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:05.858174 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h_odh-ai-gateway-infra(f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" Oct 31 01:15:05.859444 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:05.859428 5978 scope.go:122] "RemoveContainer" containerID="2bbcd77f9fdf3867c23a555bef67ca0775e25a5f330ab74e51b4c935d22e1c9c" Oct 31 01:15:05.859668 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:05.859651 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r_odh-ai-gateway-infra(4c897718-f2e1-4b61-b574-0b6a7701e078)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" Oct 31 01:15:06.428322 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:06.428286 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" path="/var/lib/kubelet/pods/8ec2edd6-67f1-4bd4-b4ab-b64a389ead00/volumes" Oct 31 01:15:07.384951 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:07.384918 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:08.944916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:08.944874 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:10.578049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:10.578016 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:10.751769 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:10.751727 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:12.380022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:12.379977 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:12.790252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:12.790097 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:13.003981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:13.003934 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:13.133630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:13.133595 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm"] Oct 31 01:15:13.133893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:13.133857 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" containerID="cri-o://a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186" gracePeriod=30 Oct 31 01:15:13.201348 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:13.201301 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:13.735607 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:13.735555 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:13.995175 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:13.995103 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825"] Oct 31 01:15:13.996084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:13.996058 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" containerName="maas-api" Oct 31 01:15:13.996084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:13.996081 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" containerName="maas-api" Oct 31 01:15:13.996246 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:13.996237 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec2edd6-67f1-4bd4-b4ab-b64a389ead00" containerName="maas-api" Oct 31 01:15:14.001155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.001138 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.013763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.013744 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:14.027771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.027745 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825"] Oct 31 01:15:14.096027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.095988 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19295660-adf8-4467-873f-110938eee54d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.096198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.096174 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv784\" (UniqueName: \"kubernetes.io/projected/19295660-adf8-4467-873f-110938eee54d-kube-api-access-qv784\") pod \"maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.096284 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.096215 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19295660-adf8-4467-873f-110938eee54d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.197584 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.197551 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19295660-adf8-4467-873f-110938eee54d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.197734 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.197679 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv784\" (UniqueName: \"kubernetes.io/projected/19295660-adf8-4467-873f-110938eee54d-kube-api-access-qv784\") pod \"maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.197734 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.197719 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19295660-adf8-4467-873f-110938eee54d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.198267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.198246 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19295660-adf8-4467-873f-110938eee54d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.200050 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.200031 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19295660-adf8-4467-873f-110938eee54d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.222116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.222082 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv784\" (UniqueName: \"kubernetes.io/projected/19295660-adf8-4467-873f-110938eee54d-kube-api-access-qv784\") pod \"maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.316106 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.316021 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:14.373666 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.373591 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:14.419212 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.419031 5978 scope.go:122] "RemoveContainer" containerID="368955c666d721934de01864f83df2a51101ce97f0f47bb8375f5d8c88d0607c" Oct 31 01:15:14.419212 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.419176 5978 scope.go:122] "RemoveContainer" containerID="20a1c75eb3780036665f4f93028139b7ecb8d276baa90b2fedf8b94e8e6a6a76" Oct 31 01:15:14.663159 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.663133 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825"] Oct 31 01:15:14.663862 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:15:14.663838 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19295660_adf8_4467_873f_110938eee54d.slice/crio-3f188e6924b4fdd0df4a5c48a7ace21ada06a77e277e43e414bbd6b049aa7056 WatchSource:0}: Error finding container 3f188e6924b4fdd0df4a5c48a7ace21ada06a77e277e43e414bbd6b049aa7056: Status 404 returned error can't find the container with id 3f188e6924b4fdd0df4a5c48a7ace21ada06a77e277e43e414bbd6b049aa7056 Oct 31 01:15:14.939384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.939351 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" event={"ID":"19295660-adf8-4467-873f-110938eee54d","Type":"ContainerStarted","Data":"3f188e6924b4fdd0df4a5c48a7ace21ada06a77e277e43e414bbd6b049aa7056"} Oct 31 01:15:14.941412 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.941388 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" event={"ID":"5ee0a45f-f22a-4f80-afac-4b1a55d8d386","Type":"ContainerStarted","Data":"007b0881bfc431fe3c9d6b0ce9dd782e0383e8c4c5f0d0dce2bfa98e7af46f6c"} Oct 31 01:15:14.943362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:14.943340 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" event={"ID":"c27563c1-b719-44fa-9cb6-fb1571c7f7d3","Type":"ContainerStarted","Data":"c3e1ceb2611d2fa4bc73f8c78e1d08ab934437b0642e0eef6680c53c7222778c"} Oct 31 01:15:15.953158 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.953073 5978 generic.go:356] "Generic (PLEG): container finished" podID="19295660-adf8-4467-873f-110938eee54d" containerID="ff4388c41f6e08e5e2dfe42731798a60187efca26fef5ac597d9288905010c6b" exitCode=0 Oct 31 01:15:15.953651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.953169 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" event={"ID":"19295660-adf8-4467-873f-110938eee54d","Type":"ContainerDied","Data":"ff4388c41f6e08e5e2dfe42731798a60187efca26fef5ac597d9288905010c6b"} Oct 31 01:15:15.955495 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.955467 5978 generic.go:356] "Generic (PLEG): container finished" podID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerID="007b0881bfc431fe3c9d6b0ce9dd782e0383e8c4c5f0d0dce2bfa98e7af46f6c" exitCode=60 Oct 31 01:15:15.955641 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.955490 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" event={"ID":"5ee0a45f-f22a-4f80-afac-4b1a55d8d386","Type":"ContainerDied","Data":"007b0881bfc431fe3c9d6b0ce9dd782e0383e8c4c5f0d0dce2bfa98e7af46f6c"} Oct 31 01:15:15.955641 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.955553 5978 scope.go:122] "RemoveContainer" containerID="20a1c75eb3780036665f4f93028139b7ecb8d276baa90b2fedf8b94e8e6a6a76" Oct 31 01:15:15.955971 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.955957 5978 scope.go:122] "RemoveContainer" containerID="007b0881bfc431fe3c9d6b0ce9dd782e0383e8c4c5f0d0dce2bfa98e7af46f6c" Oct 31 01:15:15.956363 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:15.956339 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj_odh-ai-gateway-infra(5ee0a45f-f22a-4f80-afac-4b1a55d8d386)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" Oct 31 01:15:15.958363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.958342 5978 generic.go:356] "Generic (PLEG): container finished" podID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerID="c3e1ceb2611d2fa4bc73f8c78e1d08ab934437b0642e0eef6680c53c7222778c" exitCode=60 Oct 31 01:15:15.958481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.958413 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" event={"ID":"c27563c1-b719-44fa-9cb6-fb1571c7f7d3","Type":"ContainerDied","Data":"c3e1ceb2611d2fa4bc73f8c78e1d08ab934437b0642e0eef6680c53c7222778c"} Oct 31 01:15:15.958902 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.958880 5978 scope.go:122] "RemoveContainer" containerID="c3e1ceb2611d2fa4bc73f8c78e1d08ab934437b0642e0eef6680c53c7222778c" Oct 31 01:15:15.959289 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:15.959266 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4_odh-ai-gateway-infra(c27563c1-b719-44fa-9cb6-fb1571c7f7d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" Oct 31 01:15:15.978544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.978525 5978 scope.go:122] "RemoveContainer" containerID="368955c666d721934de01864f83df2a51101ce97f0f47bb8375f5d8c88d0607c" Oct 31 01:15:15.995869 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:15.995843 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj"] Oct 31 01:15:16.033579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:16.033551 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4"] Oct 31 01:15:16.206349 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:16.206071 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:16.418670 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:16.418639 5978 scope.go:122] "RemoveContainer" containerID="f97b8e59de08aff284e1bca92a829c2d9c73737da52b37c7bd5b81ee7dccfad4" Oct 31 01:15:16.971448 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:16.971407 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" event={"ID":"ba5e3387-cb76-4cf6-a796-2b2cfebb634a","Type":"ContainerStarted","Data":"5bffd1a10e6a4dd6f8b763ec8fcf9d599d2e0401d187c8dff7c4b200c4583ca7"} Oct 31 01:15:17.196374 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.196237 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:17.249816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.249790 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:17.253812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.253788 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.354728 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-maas-api-service-ca\") pod \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\" (UID: \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\") " Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.354886 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19295660-adf8-4467-873f-110938eee54d-kube-api-access-qv784\" (UniqueName: \"kubernetes.io/projected/19295660-adf8-4467-873f-110938eee54d-kube-api-access-qv784\") pod \"19295660-adf8-4467-873f-110938eee54d\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.355185 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-maas-api-service-ca" pod "c27563c1-b719-44fa-9cb6-fb1571c7f7d3" (UID: "c27563c1-b719-44fa-9cb6-fb1571c7f7d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.355512 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-kube-api-access-jqpfn\" (UniqueName: \"kubernetes.io/projected/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-kube-api-access-jqpfn\") pod \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\" (UID: \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\") " Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.355553 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19295660-adf8-4467-873f-110938eee54d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19295660-adf8-4467-873f-110938eee54d-maas-api-openshift-ca\") pod \"19295660-adf8-4467-873f-110938eee54d\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.355612 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-maas-api-service-ca\") pod \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\" (UID: \"5ee0a45f-f22a-4f80-afac-4b1a55d8d386\") " Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.355837 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-kube-api-access-gd2bc\" (UniqueName: \"kubernetes.io/projected/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-kube-api-access-gd2bc\") pod \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\" (UID: \"c27563c1-b719-44fa-9cb6-fb1571c7f7d3\") " Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.356250 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-maas-api-service-ca" pod "5ee0a45f-f22a-4f80-afac-4b1a55d8d386" (UID: "5ee0a45f-f22a-4f80-afac-4b1a55d8d386"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.356320 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19295660-adf8-4467-873f-110938eee54d-maas-api-openshift-ca" pod "19295660-adf8-4467-873f-110938eee54d" (UID: "19295660-adf8-4467-873f-110938eee54d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:17.357537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.356625 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19295660-adf8-4467-873f-110938eee54d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19295660-adf8-4467-873f-110938eee54d-maas-api-xks-ca\") pod \"19295660-adf8-4467-873f-110938eee54d\" (UID: \"19295660-adf8-4467-873f-110938eee54d\") " Oct 31 01:15:17.358238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.357663 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19295660-adf8-4467-873f-110938eee54d-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:17.358238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.357686 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:17.358238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.357701 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:17.360896 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.360869 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19295660-adf8-4467-873f-110938eee54d-maas-api-xks-ca" pod "19295660-adf8-4467-873f-110938eee54d" (UID: "19295660-adf8-4467-873f-110938eee54d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:17.361042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.360944 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-kube-api-access-jqpfn" pod "5ee0a45f-f22a-4f80-afac-4b1a55d8d386" (UID: "5ee0a45f-f22a-4f80-afac-4b1a55d8d386"). InnerVolumeSpecName "kube-api-access-jqpfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:17.361042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.360961 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-kube-api-access-gd2bc" pod "c27563c1-b719-44fa-9cb6-fb1571c7f7d3" (UID: "c27563c1-b719-44fa-9cb6-fb1571c7f7d3"). InnerVolumeSpecName "kube-api-access-gd2bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:17.361042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.361036 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19295660-adf8-4467-873f-110938eee54d-kube-api-access-qv784" pod "19295660-adf8-4467-873f-110938eee54d" (UID: "19295660-adf8-4467-873f-110938eee54d"). InnerVolumeSpecName "kube-api-access-qv784". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:17.418383 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.418356 5978 scope.go:122] "RemoveContainer" containerID="7edc18fa98c6ef146f6dba70f57c782b942e32221010522b7e3365d327b8687f" Oct 31 01:15:17.418577 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.418551 5978 scope.go:122] "RemoveContainer" containerID="2bbcd77f9fdf3867c23a555bef67ca0775e25a5f330ab74e51b4c935d22e1c9c" Oct 31 01:15:17.418702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.418628 5978 scope.go:122] "RemoveContainer" containerID="7c78eeb48d7bc88ab1f6af0b113f7876f7b8ca528a0d165b393b2f879b74ba38" Oct 31 01:15:17.460150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.460118 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv784\" (UniqueName: \"kubernetes.io/projected/19295660-adf8-4467-873f-110938eee54d-kube-api-access-qv784\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:17.460261 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.460152 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqpfn\" (UniqueName: \"kubernetes.io/projected/5ee0a45f-f22a-4f80-afac-4b1a55d8d386-kube-api-access-jqpfn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:17.460261 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.460169 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd2bc\" (UniqueName: \"kubernetes.io/projected/c27563c1-b719-44fa-9cb6-fb1571c7f7d3-kube-api-access-gd2bc\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:17.460261 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.460186 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19295660-adf8-4467-873f-110938eee54d-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:17.982358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.982277 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" Oct 31 01:15:17.982358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.982315 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825" event={"ID":"19295660-adf8-4467-873f-110938eee54d","Type":"ContainerDied","Data":"3f188e6924b4fdd0df4a5c48a7ace21ada06a77e277e43e414bbd6b049aa7056"} Oct 31 01:15:17.982848 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.982366 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f188e6924b4fdd0df4a5c48a7ace21ada06a77e277e43e414bbd6b049aa7056" Oct 31 01:15:17.984561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.984527 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" event={"ID":"8620a009-6b7c-4e05-a7e6-02ce8ea65956","Type":"ContainerStarted","Data":"273b4ea8026ab680e064c2b10e1b3373de47314ed025263432fadd00b6b6d7b4"} Oct 31 01:15:17.986170 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.986150 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" Oct 31 01:15:17.986290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.986179 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj" event={"ID":"5ee0a45f-f22a-4f80-afac-4b1a55d8d386","Type":"ContainerDied","Data":"8bc59af517db7ac86539c584faece5f8c20440568abedb20aec788f72fbe09bf"} Oct 31 01:15:17.986290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.986234 5978 scope.go:122] "RemoveContainer" containerID="007b0881bfc431fe3c9d6b0ce9dd782e0383e8c4c5f0d0dce2bfa98e7af46f6c" Oct 31 01:15:17.987900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.987876 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" Oct 31 01:15:17.987900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.987894 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4" event={"ID":"c27563c1-b719-44fa-9cb6-fb1571c7f7d3","Type":"ContainerDied","Data":"82459151ec4c84026e1bf57f615db8281c568f31f6a34ad8b815e50642f262c7"} Oct 31 01:15:17.990153 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.990129 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" event={"ID":"4c897718-f2e1-4b61-b574-0b6a7701e078","Type":"ContainerStarted","Data":"b9c46eef28caeb406c1d4acfa8ff691be99e25c35ee78a481a7491d1b4cd4c96"} Oct 31 01:15:17.993129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.992321 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" event={"ID":"b968d397-c0cc-48aa-a6fd-591bcb7c551e","Type":"ContainerStarted","Data":"14c0889f81ccb652ab72a490ed39e293d9d1c5a8186a9da7585efdb491485980"} Oct 31 01:15:17.995037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.995012 5978 generic.go:356] "Generic (PLEG): container finished" podID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerID="5bffd1a10e6a4dd6f8b763ec8fcf9d599d2e0401d187c8dff7c4b200c4583ca7" exitCode=60 Oct 31 01:15:17.995127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.995038 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" event={"ID":"ba5e3387-cb76-4cf6-a796-2b2cfebb634a","Type":"ContainerDied","Data":"5bffd1a10e6a4dd6f8b763ec8fcf9d599d2e0401d187c8dff7c4b200c4583ca7"} Oct 31 01:15:17.995426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:17.995411 5978 scope.go:122] "RemoveContainer" containerID="5bffd1a10e6a4dd6f8b763ec8fcf9d599d2e0401d187c8dff7c4b200c4583ca7" Oct 31 01:15:17.995749 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:17.995728 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d_odh-ai-gateway-infra(ba5e3387-cb76-4cf6-a796-2b2cfebb634a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" Oct 31 01:15:18.008286 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.008263 5978 scope.go:122] "RemoveContainer" containerID="c3e1ceb2611d2fa4bc73f8c78e1d08ab934437b0642e0eef6680c53c7222778c" Oct 31 01:15:18.018108 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.018086 5978 scope.go:122] "RemoveContainer" containerID="f97b8e59de08aff284e1bca92a829c2d9c73737da52b37c7bd5b81ee7dccfad4" Oct 31 01:15:18.064362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.064327 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d"] Oct 31 01:15:18.103522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.103482 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4"] Oct 31 01:15:18.127290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.127257 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-671959-29890155-f5sk4"] Oct 31 01:15:18.269544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.267281 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:18.335518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.335296 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj"] Oct 31 01:15:18.337190 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.337159 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-735123-29890155-ftdsj"] Oct 31 01:15:18.428181 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.428152 5978 scope.go:122] "RemoveContainer" containerID="e270deb074304d1d703434fab2c4f0ac1483ea17152a74104894fd4a3587537c" Oct 31 01:15:18.432535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.432503 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" path="/var/lib/kubelet/pods/5ee0a45f-f22a-4f80-afac-4b1a55d8d386/volumes" Oct 31 01:15:18.433110 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.433089 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" path="/var/lib/kubelet/pods/c27563c1-b719-44fa-9cb6-fb1571c7f7d3/volumes" Oct 31 01:15:18.664662 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.664629 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:15:18.780859 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.780828 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-istio-token\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-istio-token\") pod \"4994c0aa-deec-4f24-8422-e5be874c350c\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " Oct 31 01:15:18.781040 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781008 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-kube-api-access-975j7\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-kube-api-access-975j7\") pod \"4994c0aa-deec-4f24-8422-e5be874c350c\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " Oct 31 01:15:18.781107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781045 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-certs\") pod \"4994c0aa-deec-4f24-8422-e5be874c350c\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " Oct 31 01:15:18.781195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781180 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-credential-socket\") pod \"4994c0aa-deec-4f24-8422-e5be874c350c\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " Oct 31 01:15:18.781281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781221 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4994c0aa-deec-4f24-8422-e5be874c350c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4994c0aa-deec-4f24-8422-e5be874c350c-istiod-ca-cert\") pod \"4994c0aa-deec-4f24-8422-e5be874c350c\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " Oct 31 01:15:18.781342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781316 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-data\") pod \"4994c0aa-deec-4f24-8422-e5be874c350c\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " Oct 31 01:15:18.781433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781375 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-envoy\") pod \"4994c0aa-deec-4f24-8422-e5be874c350c\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " Oct 31 01:15:18.781433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781388 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-certs" pod "4994c0aa-deec-4f24-8422-e5be874c350c" (UID: "4994c0aa-deec-4f24-8422-e5be874c350c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:18.781606 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781438 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4994c0aa-deec-4f24-8422-e5be874c350c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4994c0aa-deec-4f24-8422-e5be874c350c-istio-podinfo\") pod \"4994c0aa-deec-4f24-8422-e5be874c350c\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " Oct 31 01:15:18.781606 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781482 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-socket\") pod \"4994c0aa-deec-4f24-8422-e5be874c350c\" (UID: \"4994c0aa-deec-4f24-8422-e5be874c350c\") " Oct 31 01:15:18.781606 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781508 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-credential-socket" pod "4994c0aa-deec-4f24-8422-e5be874c350c" (UID: "4994c0aa-deec-4f24-8422-e5be874c350c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:18.781919 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781692 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-data" pod "4994c0aa-deec-4f24-8422-e5be874c350c" (UID: "4994c0aa-deec-4f24-8422-e5be874c350c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:18.781919 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781711 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4994c0aa-deec-4f24-8422-e5be874c350c-istiod-ca-cert" pod "4994c0aa-deec-4f24-8422-e5be874c350c" (UID: "4994c0aa-deec-4f24-8422-e5be874c350c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:18.781919 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.781888 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-socket" pod "4994c0aa-deec-4f24-8422-e5be874c350c" (UID: "4994c0aa-deec-4f24-8422-e5be874c350c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:18.782256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.782225 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:18.782256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.782251 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:18.782413 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.782265 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4994c0aa-deec-4f24-8422-e5be874c350c-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:18.782413 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.782277 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:18.782413 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.782285 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:18.783708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.783683 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-kube-api-access-975j7" pod "4994c0aa-deec-4f24-8422-e5be874c350c" (UID: "4994c0aa-deec-4f24-8422-e5be874c350c"). InnerVolumeSpecName "kube-api-access-975j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:18.784136 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.784115 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-istio-token" pod "4994c0aa-deec-4f24-8422-e5be874c350c" (UID: "4994c0aa-deec-4f24-8422-e5be874c350c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:18.785422 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.785395 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-envoy" pod "4994c0aa-deec-4f24-8422-e5be874c350c" (UID: "4994c0aa-deec-4f24-8422-e5be874c350c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:18.785532 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.785470 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4994c0aa-deec-4f24-8422-e5be874c350c-istio-podinfo" pod "4994c0aa-deec-4f24-8422-e5be874c350c" (UID: "4994c0aa-deec-4f24-8422-e5be874c350c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:18.809640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.809604 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:18.883467 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.883411 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-975j7\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-kube-api-access-975j7\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:18.883634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.883447 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4994c0aa-deec-4f24-8422-e5be874c350c-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:18.883634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.883497 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4994c0aa-deec-4f24-8422-e5be874c350c-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:18.883634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:18.883506 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4994c0aa-deec-4f24-8422-e5be874c350c-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.007344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.007313 5978 generic.go:356] "Generic (PLEG): container finished" podID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerID="273b4ea8026ab680e064c2b10e1b3373de47314ed025263432fadd00b6b6d7b4" exitCode=60 Oct 31 01:15:19.007924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.007383 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" event={"ID":"8620a009-6b7c-4e05-a7e6-02ce8ea65956","Type":"ContainerDied","Data":"273b4ea8026ab680e064c2b10e1b3373de47314ed025263432fadd00b6b6d7b4"} Oct 31 01:15:19.007924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.007520 5978 scope.go:122] "RemoveContainer" containerID="7c78eeb48d7bc88ab1f6af0b113f7876f7b8ca528a0d165b393b2f879b74ba38" Oct 31 01:15:19.008044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.007970 5978 scope.go:122] "RemoveContainer" containerID="273b4ea8026ab680e064c2b10e1b3373de47314ed025263432fadd00b6b6d7b4" Oct 31 01:15:19.009064 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:19.008362 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw_odh-ai-gateway-infra(8620a009-6b7c-4e05-a7e6-02ce8ea65956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" Oct 31 01:15:19.013986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.012261 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" event={"ID":"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7","Type":"ContainerStarted","Data":"0a8d5bc41a3421dc1b00fe26da8cc1f94d1a298bd275208d8ba62e5c467403e9"} Oct 31 01:15:19.016131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.016079 5978 generic.go:356] "Generic (PLEG): container finished" podID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerID="b9c46eef28caeb406c1d4acfa8ff691be99e25c35ee78a481a7491d1b4cd4c96" exitCode=60 Oct 31 01:15:19.016283 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.016209 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" event={"ID":"4c897718-f2e1-4b61-b574-0b6a7701e078","Type":"ContainerDied","Data":"b9c46eef28caeb406c1d4acfa8ff691be99e25c35ee78a481a7491d1b4cd4c96"} Oct 31 01:15:19.016717 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.016691 5978 scope.go:122] "RemoveContainer" containerID="b9c46eef28caeb406c1d4acfa8ff691be99e25c35ee78a481a7491d1b4cd4c96" Oct 31 01:15:19.017156 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:19.017127 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r_odh-ai-gateway-infra(4c897718-f2e1-4b61-b574-0b6a7701e078)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" Oct 31 01:15:19.019407 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.019379 5978 generic.go:356] "Generic (PLEG): container finished" podID="4994c0aa-deec-4f24-8422-e5be874c350c" containerID="a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186" exitCode=0 Oct 31 01:15:19.019554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.019423 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" Oct 31 01:15:19.019554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.019489 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" event={"ID":"4994c0aa-deec-4f24-8422-e5be874c350c","Type":"ContainerDied","Data":"a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186"} Oct 31 01:15:19.019554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.019547 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm" event={"ID":"4994c0aa-deec-4f24-8422-e5be874c350c","Type":"ContainerDied","Data":"d00bd8568004f63831338f0205abf1f52484fd7461e3ab7bac2db74786c2e1c0"} Oct 31 01:15:19.021938 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.021911 5978 generic.go:356] "Generic (PLEG): container finished" podID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerID="14c0889f81ccb652ab72a490ed39e293d9d1c5a8186a9da7585efdb491485980" exitCode=60 Oct 31 01:15:19.022066 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.021956 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" event={"ID":"b968d397-c0cc-48aa-a6fd-591bcb7c551e","Type":"ContainerDied","Data":"14c0889f81ccb652ab72a490ed39e293d9d1c5a8186a9da7585efdb491485980"} Oct 31 01:15:19.022443 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.022423 5978 scope.go:122] "RemoveContainer" containerID="14c0889f81ccb652ab72a490ed39e293d9d1c5a8186a9da7585efdb491485980" Oct 31 01:15:19.022997 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:19.022952 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc_odh-ai-gateway-infra(b968d397-c0cc-48aa-a6fd-591bcb7c551e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" Oct 31 01:15:19.024658 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.024618 5978 scope.go:122] "RemoveContainer" containerID="2bbcd77f9fdf3867c23a555bef67ca0775e25a5f330ab74e51b4c935d22e1c9c" Oct 31 01:15:19.043008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.042979 5978 scope.go:122] "RemoveContainer" containerID="a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186" Oct 31 01:15:19.057584 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.057550 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw"] Oct 31 01:15:19.060521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.060497 5978 scope.go:122] "RemoveContainer" containerID="c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf" Oct 31 01:15:19.076840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.073398 5978 scope.go:122] "RemoveContainer" containerID="a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186" Oct 31 01:15:19.077029 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:19.077005 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186\": container with ID starting with a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186 not found: ID does not exist" containerID="a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186" Oct 31 01:15:19.077109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.077046 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186"} err="failed to get container status \"a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186\": rpc error: code = NotFound desc = could not find container \"a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186\": container with ID starting with a3f797f14460b8e341b03b6d5f8e405c4810716f90134e492ee8231c27412186 not found: ID does not exist" Oct 31 01:15:19.077109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.077078 5978 scope.go:122] "RemoveContainer" containerID="c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf" Oct 31 01:15:19.077382 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:19.077364 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf\": container with ID starting with c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf not found: ID does not exist" containerID="c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf" Oct 31 01:15:19.077447 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.077391 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf"} err="failed to get container status \"c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf\": rpc error: code = NotFound desc = could not find container \"c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf\": container with ID starting with c79a8f9697a662b1a762e39e9d893626be30889d45dc82ff20d1a45c181b84cf not found: ID does not exist" Oct 31 01:15:19.077447 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.077412 5978 scope.go:122] "RemoveContainer" containerID="7edc18fa98c6ef146f6dba70f57c782b942e32221010522b7e3365d327b8687f" Oct 31 01:15:19.189123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.189092 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:19.228668 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.228629 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc"] Oct 31 01:15:19.247760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.247718 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r"] Oct 31 01:15:19.290358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.290321 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-kube-api-access-7njhs\" (UniqueName: \"kubernetes.io/projected/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-kube-api-access-7njhs\") pod \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\" (UID: \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\") " Oct 31 01:15:19.290608 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.290542 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-maas-api-service-ca\") pod \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\" (UID: \"ba5e3387-cb76-4cf6-a796-2b2cfebb634a\") " Oct 31 01:15:19.290993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.290962 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-maas-api-service-ca" pod "ba5e3387-cb76-4cf6-a796-2b2cfebb634a" (UID: "ba5e3387-cb76-4cf6-a796-2b2cfebb634a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.291321 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.291297 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:19.293154 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.293121 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-kube-api-access-7njhs" pod "ba5e3387-cb76-4cf6-a796-2b2cfebb634a" (UID: "ba5e3387-cb76-4cf6-a796-2b2cfebb634a"). InnerVolumeSpecName "kube-api-access-7njhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.333579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.333534 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm"] Oct 31 01:15:19.340756 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.340710 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-75ead8-openshift-default-65576bc997-jm6sm"] Oct 31 01:15:19.392930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:19.392891 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7njhs\" (UniqueName: \"kubernetes.io/projected/ba5e3387-cb76-4cf6-a796-2b2cfebb634a-kube-api-access-7njhs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.038626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.037020 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" Oct 31 01:15:20.038626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.037095 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d" event={"ID":"ba5e3387-cb76-4cf6-a796-2b2cfebb634a","Type":"ContainerDied","Data":"53ba9a990f169af80ce306da7066d805e8c179cc6cf8931a17e9c80dc629f1ae"} Oct 31 01:15:20.038626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.037170 5978 scope.go:122] "RemoveContainer" containerID="5bffd1a10e6a4dd6f8b763ec8fcf9d599d2e0401d187c8dff7c4b200c4583ca7" Oct 31 01:15:20.190675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.188364 5978 generic.go:356] "Generic (PLEG): container finished" podID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerID="0a8d5bc41a3421dc1b00fe26da8cc1f94d1a298bd275208d8ba62e5c467403e9" exitCode=60 Oct 31 01:15:20.190675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.188667 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" event={"ID":"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7","Type":"ContainerDied","Data":"0a8d5bc41a3421dc1b00fe26da8cc1f94d1a298bd275208d8ba62e5c467403e9"} Oct 31 01:15:20.190675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.188741 5978 scope.go:122] "RemoveContainer" containerID="e270deb074304d1d703434fab2c4f0ac1483ea17152a74104894fd4a3587537c" Oct 31 01:15:20.190675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.189263 5978 scope.go:122] "RemoveContainer" containerID="0a8d5bc41a3421dc1b00fe26da8cc1f94d1a298bd275208d8ba62e5c467403e9" Oct 31 01:15:20.190675 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:20.189728 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h_odh-ai-gateway-infra(f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" Oct 31 01:15:20.208358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.208325 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d"] Oct 31 01:15:20.213943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.213901 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-714f8f-29890155-6pf9d"] Oct 31 01:15:20.299419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.299320 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h"] Oct 31 01:15:20.374692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.374611 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:20.393573 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.393548 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:20.454095 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.454009 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" path="/var/lib/kubelet/pods/4994c0aa-deec-4f24-8422-e5be874c350c/volumes" Oct 31 01:15:20.461757 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.454773 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" path="/var/lib/kubelet/pods/ba5e3387-cb76-4cf6-a796-2b2cfebb634a/volumes" Oct 31 01:15:20.527701 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.527657 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b968d397-c0cc-48aa-a6fd-591bcb7c551e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b968d397-c0cc-48aa-a6fd-591bcb7c551e-maas-api-service-ca\") pod \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\" (UID: \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\") " Oct 31 01:15:20.527944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.527925 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b968d397-c0cc-48aa-a6fd-591bcb7c551e-kube-api-access-h5xxs\" (UniqueName: \"kubernetes.io/projected/b968d397-c0cc-48aa-a6fd-591bcb7c551e-kube-api-access-h5xxs\") pod \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\" (UID: \"b968d397-c0cc-48aa-a6fd-591bcb7c551e\") " Oct 31 01:15:20.529436 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.529393 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b968d397-c0cc-48aa-a6fd-591bcb7c551e-maas-api-service-ca" pod "b968d397-c0cc-48aa-a6fd-591bcb7c551e" (UID: "b968d397-c0cc-48aa-a6fd-591bcb7c551e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.540908 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.540867 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b968d397-c0cc-48aa-a6fd-591bcb7c551e-kube-api-access-h5xxs" pod "b968d397-c0cc-48aa-a6fd-591bcb7c551e" (UID: "b968d397-c0cc-48aa-a6fd-591bcb7c551e"). InnerVolumeSpecName "kube-api-access-h5xxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.595448 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.595420 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:20.597902 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.597879 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:20.630375 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.630327 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b968d397-c0cc-48aa-a6fd-591bcb7c551e-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.630375 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.630376 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5xxs\" (UniqueName: \"kubernetes.io/projected/b968d397-c0cc-48aa-a6fd-591bcb7c551e-kube-api-access-h5xxs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.730954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.730917 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c897718-f2e1-4b61-b574-0b6a7701e078-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c897718-f2e1-4b61-b574-0b6a7701e078-maas-api-service-ca\") pod \"4c897718-f2e1-4b61-b574-0b6a7701e078\" (UID: \"4c897718-f2e1-4b61-b574-0b6a7701e078\") " Oct 31 01:15:20.731153 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.731001 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8620a009-6b7c-4e05-a7e6-02ce8ea65956-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8620a009-6b7c-4e05-a7e6-02ce8ea65956-maas-api-service-ca\") pod \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\" (UID: \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\") " Oct 31 01:15:20.731153 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.731054 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8620a009-6b7c-4e05-a7e6-02ce8ea65956-kube-api-access-bz5kd\" (UniqueName: \"kubernetes.io/projected/8620a009-6b7c-4e05-a7e6-02ce8ea65956-kube-api-access-bz5kd\") pod \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\" (UID: \"8620a009-6b7c-4e05-a7e6-02ce8ea65956\") " Oct 31 01:15:20.731291 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.731234 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c897718-f2e1-4b61-b574-0b6a7701e078-kube-api-access-b52ft\" (UniqueName: \"kubernetes.io/projected/4c897718-f2e1-4b61-b574-0b6a7701e078-kube-api-access-b52ft\") pod \"4c897718-f2e1-4b61-b574-0b6a7701e078\" (UID: \"4c897718-f2e1-4b61-b574-0b6a7701e078\") " Oct 31 01:15:20.731354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.731329 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c897718-f2e1-4b61-b574-0b6a7701e078-maas-api-service-ca" pod "4c897718-f2e1-4b61-b574-0b6a7701e078" (UID: "4c897718-f2e1-4b61-b574-0b6a7701e078"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.731508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.731434 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8620a009-6b7c-4e05-a7e6-02ce8ea65956-maas-api-service-ca" pod "8620a009-6b7c-4e05-a7e6-02ce8ea65956" (UID: "8620a009-6b7c-4e05-a7e6-02ce8ea65956"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.732053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.732027 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8620a009-6b7c-4e05-a7e6-02ce8ea65956-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.732298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.732279 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c897718-f2e1-4b61-b574-0b6a7701e078-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.733767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.733739 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8620a009-6b7c-4e05-a7e6-02ce8ea65956-kube-api-access-bz5kd" pod "8620a009-6b7c-4e05-a7e6-02ce8ea65956" (UID: "8620a009-6b7c-4e05-a7e6-02ce8ea65956"). InnerVolumeSpecName "kube-api-access-bz5kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.739909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.739877 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c897718-f2e1-4b61-b574-0b6a7701e078-kube-api-access-b52ft" pod "4c897718-f2e1-4b61-b574-0b6a7701e078" (UID: "4c897718-f2e1-4b61-b574-0b6a7701e078"). InnerVolumeSpecName "kube-api-access-b52ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.834030 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.833984 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz5kd\" (UniqueName: \"kubernetes.io/projected/8620a009-6b7c-4e05-a7e6-02ce8ea65956-kube-api-access-bz5kd\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:20.834030 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:20.834020 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b52ft\" (UniqueName: \"kubernetes.io/projected/4c897718-f2e1-4b61-b574-0b6a7701e078-kube-api-access-b52ft\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:21.210139 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.210109 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" Oct 31 01:15:21.210596 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.210154 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r" event={"ID":"4c897718-f2e1-4b61-b574-0b6a7701e078","Type":"ContainerDied","Data":"037e2b0d1f7e09f1df904b2e0051e6807fe20c50334ea695e9682b49c2dd4667"} Oct 31 01:15:21.210596 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.210230 5978 scope.go:122] "RemoveContainer" containerID="b9c46eef28caeb406c1d4acfa8ff691be99e25c35ee78a481a7491d1b4cd4c96" Oct 31 01:15:21.216818 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.216771 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" Oct 31 01:15:21.216818 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.216796 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc" event={"ID":"b968d397-c0cc-48aa-a6fd-591bcb7c551e","Type":"ContainerDied","Data":"64e2e816bac7e9f4410153a08cd340673a8c0105b62af84c097d6911a590acae"} Oct 31 01:15:21.220290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.220007 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" event={"ID":"8620a009-6b7c-4e05-a7e6-02ce8ea65956","Type":"ContainerDied","Data":"f9ea7e87d648961599cc6ab095dc3732d3c04297248f1f4967933e7b8fdf8fce"} Oct 31 01:15:21.220290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.220203 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw" Oct 31 01:15:21.225801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.225781 5978 scope.go:122] "RemoveContainer" containerID="14c0889f81ccb652ab72a490ed39e293d9d1c5a8186a9da7585efdb491485980" Oct 31 01:15:21.255891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.255857 5978 scope.go:122] "RemoveContainer" containerID="273b4ea8026ab680e064c2b10e1b3373de47314ed025263432fadd00b6b6d7b4" Oct 31 01:15:21.285777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.285740 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw"] Oct 31 01:15:21.290048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.290020 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-328d61-29890155-nnvgw"] Oct 31 01:15:21.336283 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.336233 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc"] Oct 31 01:15:21.345525 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.340923 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be301b-29890155-wxbdc"] Oct 31 01:15:21.369787 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.369753 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r"] Oct 31 01:15:21.373502 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.373445 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e41918-29890155-rlc5r"] Oct 31 01:15:21.391763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.391734 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:21.547988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.547956 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-maas-api-service-ca\") pod \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\" (UID: \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\") " Oct 31 01:15:21.548173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.548051 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-kube-api-access-gdw2z\" (UniqueName: \"kubernetes.io/projected/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-kube-api-access-gdw2z\") pod \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\" (UID: \"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7\") " Oct 31 01:15:21.548416 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.548380 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-maas-api-service-ca" pod "f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" (UID: "f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:21.548781 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.548756 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:21.550385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.550354 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-kube-api-access-gdw2z" pod "f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" (UID: "f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7"). InnerVolumeSpecName "kube-api-access-gdw2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:21.650694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:21.650655 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdw2z\" (UniqueName: \"kubernetes.io/projected/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7-kube-api-access-gdw2z\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:22.011659 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.011615 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:22.236495 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.236422 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" Oct 31 01:15:22.237013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.236425 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h" event={"ID":"f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7","Type":"ContainerDied","Data":"3c6e5d36d478021a652b33c21f4c2f0b70095f8f8391d378de82f76e2cfe9cb5"} Oct 31 01:15:22.237013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.236620 5978 scope.go:122] "RemoveContainer" containerID="0a8d5bc41a3421dc1b00fe26da8cc1f94d1a298bd275208d8ba62e5c467403e9" Oct 31 01:15:22.307118 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.307019 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h"] Oct 31 01:15:22.310034 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.309992 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:22.315675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.315642 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9e377-29890155-fz44h"] Oct 31 01:15:22.427770 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.427738 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" path="/var/lib/kubelet/pods/4c897718-f2e1-4b61-b574-0b6a7701e078/volumes" Oct 31 01:15:22.428229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.428216 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" path="/var/lib/kubelet/pods/8620a009-6b7c-4e05-a7e6-02ce8ea65956/volumes" Oct 31 01:15:22.428672 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.428657 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" path="/var/lib/kubelet/pods/b968d397-c0cc-48aa-a6fd-591bcb7c551e/volumes" Oct 31 01:15:22.429370 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.429355 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" path="/var/lib/kubelet/pods/f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7/volumes" Oct 31 01:15:22.493809 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:22.493767 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:24.406107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:24.406019 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:24.653122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:24.653090 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9"] Oct 31 01:15:24.693655 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:24.693574 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w"] Oct 31 01:15:24.870779 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:24.870726 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:25.037766 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:25.037677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:25.748446 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:25.748405 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:25.938358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:25.938320 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:26.959067 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:26.959023 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:28.949344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:28.949302 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:29.653859 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:29.653810 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" podUID="addf314e-6c50-426e-9c41-ba911bea7b61" containerName="payload-processing" containerID="cri-o://1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636" gracePeriod=25 Oct 31 01:15:29.694594 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:29.694510 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" podUID="247397bb-656a-4a0d-b63f-02a7f742e04f" containerName="payload-pre-processing" containerID="cri-o://e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6" gracePeriod=25 Oct 31 01:15:30.761064 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:30.761026 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:32.539478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:32.535387 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:33.191380 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:33.191338 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:15:34.218629 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:34.218594 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:34.805359 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:34.805322 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" podUID="addf314e-6c50-426e-9c41-ba911bea7b61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:34.805359 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:34.805351 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" podUID="247397bb-656a-4a0d-b63f-02a7f742e04f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:35.907203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:35.907160 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:37.560108 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:37.560052 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:39.371891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:39.371855 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:41.169530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:41.169445 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:42.656817 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:42.656777 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:42.861521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:42.861476 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:43.266524 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:43.259992 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:43.476778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:43.476747 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:44.706172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:44.706133 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:44.805264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:44.805223 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" podUID="addf314e-6c50-426e-9c41-ba911bea7b61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:44.805500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:44.805224 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" podUID="247397bb-656a-4a0d-b63f-02a7f742e04f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:46.333710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:46.333668 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:47.637581 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:47.637535 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:47.793753 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:47.793707 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:48.267386 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:48.267348 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:49.331846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:49.331799 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:49.518152 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:49.518115 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:49.698711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:49.698079 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:49.978377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:49.978349 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 01:15:50.010688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.010658 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 01:15:50.028593 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.028561 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addf314e-6c50-426e-9c41-ba911bea7b61-kube-api-access-8bp4p\" (UniqueName: \"kubernetes.io/projected/addf314e-6c50-426e-9c41-ba911bea7b61-kube-api-access-8bp4p\") pod \"addf314e-6c50-426e-9c41-ba911bea7b61\" (UID: \"addf314e-6c50-426e-9c41-ba911bea7b61\") " Oct 31 01:15:50.028802 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.028605 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addf314e-6c50-426e-9c41-ba911bea7b61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf314e-6c50-426e-9c41-ba911bea7b61-plugins-config-volume\") pod \"addf314e-6c50-426e-9c41-ba911bea7b61\" (UID: \"addf314e-6c50-426e-9c41-ba911bea7b61\") " Oct 31 01:15:50.029043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.029018 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addf314e-6c50-426e-9c41-ba911bea7b61-plugins-config-volume" pod "addf314e-6c50-426e-9c41-ba911bea7b61" (UID: "addf314e-6c50-426e-9c41-ba911bea7b61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:50.029348 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.029329 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addf314e-6c50-426e-9c41-ba911bea7b61-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:50.031352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.031323 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addf314e-6c50-426e-9c41-ba911bea7b61-kube-api-access-8bp4p" pod "addf314e-6c50-426e-9c41-ba911bea7b61" (UID: "addf314e-6c50-426e-9c41-ba911bea7b61"). InnerVolumeSpecName "kube-api-access-8bp4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:50.054164 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.053997 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:50.130960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.130923 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247397bb-656a-4a0d-b63f-02a7f742e04f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247397bb-656a-4a0d-b63f-02a7f742e04f-plugins-config-volume\") pod \"247397bb-656a-4a0d-b63f-02a7f742e04f\" (UID: \"247397bb-656a-4a0d-b63f-02a7f742e04f\") " Oct 31 01:15:50.131137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.131105 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247397bb-656a-4a0d-b63f-02a7f742e04f-kube-api-access-m6xs7\" (UniqueName: \"kubernetes.io/projected/247397bb-656a-4a0d-b63f-02a7f742e04f-kube-api-access-m6xs7\") pod \"247397bb-656a-4a0d-b63f-02a7f742e04f\" (UID: \"247397bb-656a-4a0d-b63f-02a7f742e04f\") " Oct 31 01:15:50.131591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.131569 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bp4p\" (UniqueName: \"kubernetes.io/projected/addf314e-6c50-426e-9c41-ba911bea7b61-kube-api-access-8bp4p\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:50.132490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.132461 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247397bb-656a-4a0d-b63f-02a7f742e04f-plugins-config-volume" pod "247397bb-656a-4a0d-b63f-02a7f742e04f" (UID: "247397bb-656a-4a0d-b63f-02a7f742e04f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:50.133586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.133564 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247397bb-656a-4a0d-b63f-02a7f742e04f-kube-api-access-m6xs7" pod "247397bb-656a-4a0d-b63f-02a7f742e04f" (UID: "247397bb-656a-4a0d-b63f-02a7f742e04f"). InnerVolumeSpecName "kube-api-access-m6xs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:50.233511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.233395 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247397bb-656a-4a0d-b63f-02a7f742e04f-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:50.233511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.233423 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6xs7\" (UniqueName: \"kubernetes.io/projected/247397bb-656a-4a0d-b63f-02a7f742e04f-kube-api-access-m6xs7\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:15:50.472427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.472386 5978 generic.go:356] "Generic (PLEG): container finished" podID="247397bb-656a-4a0d-b63f-02a7f742e04f" containerID="e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6" exitCode=0 Oct 31 01:15:50.472996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.472443 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" Oct 31 01:15:50.472996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.472474 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" event={"ID":"247397bb-656a-4a0d-b63f-02a7f742e04f","Type":"ContainerDied","Data":"e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6"} Oct 31 01:15:50.472996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.472532 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w" event={"ID":"247397bb-656a-4a0d-b63f-02a7f742e04f","Type":"ContainerDied","Data":"f72f4575c52533ea5c5200b7aba54dfc5e00f1a2ba1a07ddb4bcaee55e87119c"} Oct 31 01:15:50.472996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.472556 5978 scope.go:122] "RemoveContainer" containerID="e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6" Oct 31 01:15:50.474262 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.474224 5978 generic.go:356] "Generic (PLEG): container finished" podID="addf314e-6c50-426e-9c41-ba911bea7b61" containerID="1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636" exitCode=0 Oct 31 01:15:50.474382 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.474276 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" Oct 31 01:15:50.474382 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.474326 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" event={"ID":"addf314e-6c50-426e-9c41-ba911bea7b61","Type":"ContainerDied","Data":"1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636"} Oct 31 01:15:50.474559 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.474380 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9" event={"ID":"addf314e-6c50-426e-9c41-ba911bea7b61","Type":"ContainerDied","Data":"f960d776a3125f869d1fe502b739e7c56f4be970afd1803ba8e470b8b00a6b0d"} Oct 31 01:15:50.484590 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.484559 5978 scope.go:122] "RemoveContainer" containerID="e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6" Oct 31 01:15:50.485025 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:50.484998 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6\": container with ID starting with e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6 not found: ID does not exist" containerID="e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6" Oct 31 01:15:50.485119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.485034 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6"} err="failed to get container status \"e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6\": rpc error: code = NotFound desc = could not find container \"e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6\": container with ID starting with e9087e03e8ebdea1867b6cf719cf005d2c77d460bba49a28553aa57972e417c6 not found: ID does not exist" Oct 31 01:15:50.485119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.485052 5978 scope.go:122] "RemoveContainer" containerID="1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636" Oct 31 01:15:50.497866 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.497840 5978 scope.go:122] "RemoveContainer" containerID="1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636" Oct 31 01:15:50.498196 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:15:50.498167 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636\": container with ID starting with 1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636 not found: ID does not exist" containerID="1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636" Oct 31 01:15:50.498270 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.498209 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636"} err="failed to get container status \"1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636\": rpc error: code = NotFound desc = could not find container \"1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636\": container with ID starting with 1d6bb63d99463a8f94b18b148c940eefcfd46142b8c54963ba88b0018ab2d636 not found: ID does not exist" Oct 31 01:15:50.521251 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.521215 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9"] Oct 31 01:15:50.530107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.530072 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0d8bae-5b464cf9d7-vlsb9"] Oct 31 01:15:50.558076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.558033 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w"] Oct 31 01:15:50.567131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.567096 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0d8bae-85c849467-dws9w"] Oct 31 01:15:50.861309 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:50.861265 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:52.427335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:52.427301 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247397bb-656a-4a0d-b63f-02a7f742e04f" path="/var/lib/kubelet/pods/247397bb-656a-4a0d-b63f-02a7f742e04f/volumes" Oct 31 01:15:52.428544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:52.427780 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addf314e-6c50-426e-9c41-ba911bea7b61" path="/var/lib/kubelet/pods/addf314e-6c50-426e-9c41-ba911bea7b61/volumes" Oct 31 01:15:52.497260 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:52.497218 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:54.307519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:54.297620 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:55.942365 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:55.942327 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:56.093001 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:56.092958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:56.257979 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:56.257882 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:57.361488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:57.361425 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:57.836284 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:57.836256 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:58.201363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:58.201277 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz"] Oct 31 01:15:58.392657 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:58.392600 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:58.531136 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:58.531014 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:58.671617 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:58.671582 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:59.548113 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:15:59.548077 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:01.505957 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:01.505923 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:02.888725 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:02.888688 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:03.059736 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.059702 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:03.260886 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.253756 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" podUID="1c4195cf-cae9-4353-8ea4-9fd607e6bf6b" containerName="payload-processing" containerID="cri-o://b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4" gracePeriod=25 Oct 31 01:16:03.323873 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:16:03.323832 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:16:03.427122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427083 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq"] Oct 31 01:16:03.427822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427799 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerName="cleanup" Oct 31 01:16:03.427822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427816 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerName="cleanup" Oct 31 01:16:03.427822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427829 5978 cpu_manager.go:414] "containerMap: removing container" podUID="247397bb-656a-4a0d-b63f-02a7f742e04f" containerName="payload-pre-processing" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427834 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247397bb-656a-4a0d-b63f-02a7f742e04f" containerName="payload-pre-processing" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427846 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427851 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427858 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427863 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427876 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427880 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427891 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427895 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427902 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427907 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427913 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427918 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427926 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427930 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427938 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427943 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427953 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427958 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427964 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427968 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427976 5978 cpu_manager.go:414] "containerMap: removing container" podUID="addf314e-6c50-426e-9c41-ba911bea7b61" containerName="payload-processing" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427980 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addf314e-6c50-426e-9c41-ba911bea7b61" containerName="payload-processing" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427988 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.427993 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428005 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428009 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428017 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428021 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428028 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428033 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428043 5978 cpu_manager.go:414] "containerMap: removing container" podUID="19295660-adf8-4467-873f-110938eee54d" containerName="revoke-keys" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428047 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19295660-adf8-4467-873f-110938eee54d" containerName="revoke-keys" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428058 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428062 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428073 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428079 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428091 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428097 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428103 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerName="cleanup" Oct 31 01:16:03.428084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428109 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428120 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428124 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428130 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428134 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428243 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="247397bb-656a-4a0d-b63f-02a7f742e04f" containerName="payload-pre-processing" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428252 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428258 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428264 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428270 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428275 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428282 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428288 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428293 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428299 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428304 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428309 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="addf314e-6c50-426e-9c41-ba911bea7b61" containerName="payload-processing" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428314 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428320 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428325 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428331 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="19295660-adf8-4467-873f-110938eee54d" containerName="revoke-keys" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428336 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428341 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerName="cleanup" Oct 31 01:16:03.430899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.428348 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerName="cleanup" Oct 31 01:16:03.432596 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.432119 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:03.450917 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.450891 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-33d44f-dockercfg-2r5fm\"" Oct 31 01:16:03.451072 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.451005 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-33d44f\"" Oct 31 01:16:03.457155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.457125 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq"] Oct 31 01:16:03.560423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.560395 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:16:03.568711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.566345 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq\" (UID: \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:03.568711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.566446 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvkxr\" (UniqueName: \"kubernetes.io/projected/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-kube-api-access-rvkxr\") pod \"payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq\" (UID: \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:03.595578 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.595544 5978 generic.go:356] "Generic (PLEG): container finished" podID="1c4195cf-cae9-4353-8ea4-9fd607e6bf6b" containerID="b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4" exitCode=0 Oct 31 01:16:03.595753 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.595601 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" Oct 31 01:16:03.595753 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.595631 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" event={"ID":"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b","Type":"ContainerDied","Data":"b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4"} Oct 31 01:16:03.595753 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.595718 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz" event={"ID":"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b","Type":"ContainerDied","Data":"143c35f7c5240ee2164fb3e180e23532a06d9e800f847e111e86531b571087f2"} Oct 31 01:16:03.595753 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.595742 5978 scope.go:122] "RemoveContainer" containerID="b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4" Oct 31 01:16:03.608321 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.608299 5978 scope.go:122] "RemoveContainer" containerID="b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4" Oct 31 01:16:03.608745 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:16:03.608719 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4\": container with ID starting with b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4 not found: ID does not exist" containerID="b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4" Oct 31 01:16:03.608852 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.608752 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4"} err="failed to get container status \"b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4\": rpc error: code = NotFound desc = could not find container \"b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4\": container with ID starting with b2d2998c74e16b4ee340b293523c37b34df78adec6a71a6e532828e5cb0488e4 not found: ID does not exist" Oct 31 01:16:03.668596 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.668560 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-kube-api-access-dpmnh\" (UniqueName: \"kubernetes.io/projected/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-kube-api-access-dpmnh\") pod \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\" (UID: \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\") " Oct 31 01:16:03.668831 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.668724 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-plugins-config-volume\") pod \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\" (UID: \"1c4195cf-cae9-4353-8ea4-9fd607e6bf6b\") " Oct 31 01:16:03.669030 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.669012 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq\" (UID: \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:03.669079 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.669054 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvkxr\" (UniqueName: \"kubernetes.io/projected/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-kube-api-access-rvkxr\") pod \"payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq\" (UID: \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:03.669122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.669090 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-plugins-config-volume" pod "1c4195cf-cae9-4353-8ea4-9fd607e6bf6b" (UID: "1c4195cf-cae9-4353-8ea4-9fd607e6bf6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:03.669294 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.669278 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:03.669991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.669966 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq\" (UID: \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:03.671150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.671121 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-kube-api-access-dpmnh" pod "1c4195cf-cae9-4353-8ea4-9fd607e6bf6b" (UID: "1c4195cf-cae9-4353-8ea4-9fd607e6bf6b"). InnerVolumeSpecName "kube-api-access-dpmnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:03.692682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.692653 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvkxr\" (UniqueName: \"kubernetes.io/projected/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-kube-api-access-rvkxr\") pod \"payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq\" (UID: \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:03.748710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.748672 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:03.771011 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.770979 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpmnh\" (UniqueName: \"kubernetes.io/projected/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b-kube-api-access-dpmnh\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:03.949444 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.949406 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz"] Oct 31 01:16:03.957736 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:03.957704 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa97ea-5bb454dbc9-mt8xz"] Oct 31 01:16:04.061772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:04.061622 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq"] Oct 31 01:16:04.064536 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:16:04.064508 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6b4292_2d3c_4c9e_80bd_18cc48ed1f66.slice/crio-7907623c4ae5117d15cd5adb1e068da72e1e73275c7a090a6f748032c98b314c WatchSource:0}: Error finding container 7907623c4ae5117d15cd5adb1e068da72e1e73275c7a090a6f748032c98b314c: Status 404 returned error can't find the container with id 7907623c4ae5117d15cd5adb1e068da72e1e73275c7a090a6f748032c98b314c Oct 31 01:16:04.178372 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:04.178340 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:04.428723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:04.428687 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4195cf-cae9-4353-8ea4-9fd607e6bf6b" path="/var/lib/kubelet/pods/1c4195cf-cae9-4353-8ea4-9fd607e6bf6b/volumes" Oct 31 01:16:04.608110 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:04.608074 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" event={"ID":"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66","Type":"ContainerStarted","Data":"70c94eceeefb08175768bf9b5b1f3b1ce3a127878a59508c49e7690c5934a4b7"} Oct 31 01:16:04.608341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:04.608138 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" event={"ID":"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66","Type":"ContainerStarted","Data":"7907623c4ae5117d15cd5adb1e068da72e1e73275c7a090a6f748032c98b314c"} Oct 31 01:16:04.608420 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:04.608372 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:04.651991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:04.651931 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" podStartSLOduration=1.651916114 podStartE2EDuration="1.651916114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:04.650888895 +0000 UTC m=+3382.349002659" watchObservedRunningTime="2026-10-31 01:16:04.651916114 +0000 UTC m=+3382.350029879" Oct 31 01:16:05.625199 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:05.625164 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:06.136329 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:06.136286 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:07.869414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:07.869374 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:10.222478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:10.221884 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:11.936932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:11.934961 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:13.535187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:13.535147 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:15.210123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:15.210085 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:15.618272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:15.618241 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:16:16.830479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:16.828677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:18.646521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:18.646438 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:20.330518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:20.330481 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:21.249430 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:21.249391 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:22.233484 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:22.216610 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:22.648226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:22.648194 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:16:22.754749 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:22.754715 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:22.881433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:22.881400 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:16:22.965116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:22.965028 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:23.077521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:23.077482 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq_1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66/payload-processing/0.log" Oct 31 01:16:23.986887 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:23.982393 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:25.732707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:25.732667 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:27.407345 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:27.406608 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:29.137913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:29.137871 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:29.304967 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:29.304931 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:30.860477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:30.859766 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:32.492182 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:32.492142 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:33.182280 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:16:33.182243 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:16:34.231775 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:34.231735 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:36.255688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:36.255649 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:36.475626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:36.475583 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:36.622512 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:36.622478 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:37.668719 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:37.668679 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:38.208047 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:38.208009 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:38.343250 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:38.343214 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825"] Oct 31 01:16:38.369374 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:38.368034 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:38.373437 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:38.373406 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0d8bae-zv825"] Oct 31 01:16:38.435663 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:38.435625 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19295660-adf8-4467-873f-110938eee54d" path="/var/lib/kubelet/pods/19295660-adf8-4467-873f-110938eee54d/volumes" Oct 31 01:16:38.530053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:38.529969 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:38.890872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:38.890834 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:39.963535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:39.963497 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:40.126685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:40.126645 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:41.134509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:41.111217 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:42.805929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:42.805893 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:44.515768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:44.515491 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:45.477032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:45.476994 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:45.630167 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:45.630134 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:45.777112 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:45.777032 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh"] Oct 31 01:16:45.777269 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:45.777252 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" containerID="cri-o://09de2e61761e4f118b452377ecdac5127c4d34945632547d8476172be1977c31" gracePeriod=30 Oct 31 01:16:45.828135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:45.828106 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:46.133560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:46.133520 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:48.349555 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:48.349469 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:48.746553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:48.746515 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:49.773440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:49.773396 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:49.945053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:49.945020 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:51.039522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.039413 5978 generic.go:356] "Generic (PLEG): container finished" podID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerID="09de2e61761e4f118b452377ecdac5127c4d34945632547d8476172be1977c31" exitCode=0 Oct 31 01:16:51.040048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.039666 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" event={"ID":"d476266c-048c-469f-9fc6-7bad2061ea9c","Type":"ContainerDied","Data":"09de2e61761e4f118b452377ecdac5127c4d34945632547d8476172be1977c31"} Oct 31 01:16:51.089546 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.089512 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 01:16:51.211654 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.211619 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-kube-api-access-6h9l8\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-kube-api-access-6h9l8\") pod \"d476266c-048c-469f-9fc6-7bad2061ea9c\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " Oct 31 01:16:51.211851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.211679 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-socket\") pod \"d476266c-048c-469f-9fc6-7bad2061ea9c\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " Oct 31 01:16:51.211851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.211704 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-certs\") pod \"d476266c-048c-469f-9fc6-7bad2061ea9c\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " Oct 31 01:16:51.211851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.211727 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-token\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-token\") pod \"d476266c-048c-469f-9fc6-7bad2061ea9c\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " Oct 31 01:16:51.211851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.211744 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d476266c-048c-469f-9fc6-7bad2061ea9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d476266c-048c-469f-9fc6-7bad2061ea9c-istiod-ca-cert\") pod \"d476266c-048c-469f-9fc6-7bad2061ea9c\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " Oct 31 01:16:51.212009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.211853 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-data\") pod \"d476266c-048c-469f-9fc6-7bad2061ea9c\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " Oct 31 01:16:51.212009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.211910 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-envoy\") pod \"d476266c-048c-469f-9fc6-7bad2061ea9c\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " Oct 31 01:16:51.212009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.211994 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-podinfo\") pod \"d476266c-048c-469f-9fc6-7bad2061ea9c\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " Oct 31 01:16:51.212111 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.212068 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-credential-socket\") pod \"d476266c-048c-469f-9fc6-7bad2061ea9c\" (UID: \"d476266c-048c-469f-9fc6-7bad2061ea9c\") " Oct 31 01:16:51.214091 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.214055 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-certs" pod "d476266c-048c-469f-9fc6-7bad2061ea9c" (UID: "d476266c-048c-469f-9fc6-7bad2061ea9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:51.214298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.214164 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-data" pod "d476266c-048c-469f-9fc6-7bad2061ea9c" (UID: "d476266c-048c-469f-9fc6-7bad2061ea9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:51.214771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.214729 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d476266c-048c-469f-9fc6-7bad2061ea9c-istiod-ca-cert" pod "d476266c-048c-469f-9fc6-7bad2061ea9c" (UID: "d476266c-048c-469f-9fc6-7bad2061ea9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:51.214908 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.214759 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-socket" pod "d476266c-048c-469f-9fc6-7bad2061ea9c" (UID: "d476266c-048c-469f-9fc6-7bad2061ea9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:51.214908 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.214893 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-credential-socket" pod "d476266c-048c-469f-9fc6-7bad2061ea9c" (UID: "d476266c-048c-469f-9fc6-7bad2061ea9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:51.215629 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.215607 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-kube-api-access-6h9l8" pod "d476266c-048c-469f-9fc6-7bad2061ea9c" (UID: "d476266c-048c-469f-9fc6-7bad2061ea9c"). InnerVolumeSpecName "kube-api-access-6h9l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:51.215727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.215626 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-envoy" pod "d476266c-048c-469f-9fc6-7bad2061ea9c" (UID: "d476266c-048c-469f-9fc6-7bad2061ea9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:51.215727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.215646 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-token" pod "d476266c-048c-469f-9fc6-7bad2061ea9c" (UID: "d476266c-048c-469f-9fc6-7bad2061ea9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:51.216858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.216843 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-podinfo" pod "d476266c-048c-469f-9fc6-7bad2061ea9c" (UID: "d476266c-048c-469f-9fc6-7bad2061ea9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:16:51.314176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.314082 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:51.314176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.314117 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:51.314176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.314129 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:51.314176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.314142 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h9l8\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-kube-api-access-6h9l8\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:51.314176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.314154 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:51.314176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.314166 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:51.314176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.314179 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:51.314587 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.314190 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d476266c-048c-469f-9fc6-7bad2061ea9c-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:51.314587 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.314203 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d476266c-048c-469f-9fc6-7bad2061ea9c-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:16:51.619675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:51.619631 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:52.048656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:52.048573 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" event={"ID":"d476266c-048c-469f-9fc6-7bad2061ea9c","Type":"ContainerDied","Data":"def0b976e7ad518f5a09369fdac763dd38f819b133f28ab200683cd5bae82d92"} Oct 31 01:16:52.048656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:52.048630 5978 scope.go:122] "RemoveContainer" containerID="09de2e61761e4f118b452377ecdac5127c4d34945632547d8476172be1977c31" Oct 31 01:16:52.049137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:52.048585 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh" Oct 31 01:16:52.096001 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:52.095958 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh"] Oct 31 01:16:52.105650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:52.105617 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0d8bae-openshift-default-6c96b68449-tgxhh"] Oct 31 01:16:52.439631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:52.439582 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" path="/var/lib/kubelet/pods/d476266c-048c-469f-9fc6-7bad2061ea9c/volumes" Oct 31 01:16:53.697881 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:53.697846 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:55.341184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:55.341143 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:56.316174 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.316141 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:56.772535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.772497 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr"] Oct 31 01:16:56.773358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773334 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerName="cleanup" Oct 31 01:16:56.773358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773358 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5e3387-cb76-4cf6-a796-2b2cfebb634a" containerName="cleanup" Oct 31 01:16:56.773591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773375 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerName="cleanup" Oct 31 01:16:56.773591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773382 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerName="cleanup" Oct 31 01:16:56.773591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773400 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" Oct 31 01:16:56.773591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773406 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" Oct 31 01:16:56.773591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773480 5978 cpu_manager.go:414] "containerMap: removing container" podUID="1c4195cf-cae9-4353-8ea4-9fd607e6bf6b" containerName="payload-processing" Oct 31 01:16:56.773591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773488 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4195cf-cae9-4353-8ea4-9fd607e6bf6b" containerName="payload-processing" Oct 31 01:16:56.773907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773662 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d73a0c-adbb-45c2-8e65-f2e7e59dc2b7" containerName="cleanup" Oct 31 01:16:56.773907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773678 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8620a009-6b7c-4e05-a7e6-02ce8ea65956" containerName="cleanup" Oct 31 01:16:56.773907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773691 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4994c0aa-deec-4f24-8422-e5be874c350c" containerName="istio-proxy" Oct 31 01:16:56.773907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773701 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee0a45f-f22a-4f80-afac-4b1a55d8d386" containerName="cleanup" Oct 31 01:16:56.773907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773711 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d476266c-048c-469f-9fc6-7bad2061ea9c" containerName="istio-proxy" Oct 31 01:16:56.773907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773720 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b968d397-c0cc-48aa-a6fd-591bcb7c551e" containerName="cleanup" Oct 31 01:16:56.773907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773730 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27563c1-b719-44fa-9cb6-fb1571c7f7d3" containerName="cleanup" Oct 31 01:16:56.773907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773739 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c897718-f2e1-4b61-b574-0b6a7701e078" containerName="cleanup" Oct 31 01:16:56.773907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.773753 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4195cf-cae9-4353-8ea4-9fd607e6bf6b" containerName="payload-processing" Oct 31 01:16:56.779704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.779679 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:56.804899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.804871 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8d251f-openshift-default-dockercfg-46zvz\"" Oct 31 01:16:56.817846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.817810 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr"] Oct 31 01:16:56.887584 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.887536 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:56.910610 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.910570 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-data\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:56.910793 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.910614 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d2df58-6860-403f-acb9-1b28d74a7791-istio-podinfo\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:56.910793 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.910731 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqq7\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-kube-api-access-jfqq7\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:56.910793 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.910767 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-envoy\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:56.910943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.910816 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-certs\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:56.910943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.910915 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-istio-token\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:56.911044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.910942 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d2df58-6860-403f-acb9-1b28d74a7791-istiod-ca-cert\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:56.911114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.911093 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-socket\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:56.911191 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:56.911175 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-credential-socket\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.023531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.014807 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqq7\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-kube-api-access-jfqq7\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.023531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.014868 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-envoy\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.023531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.014907 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-certs\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.023531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.014954 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-istio-token\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.023531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.014979 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d2df58-6860-403f-acb9-1b28d74a7791-istiod-ca-cert\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.023531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.015092 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-socket\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.023531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.015158 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-credential-socket\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.023531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.015353 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-data\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.023531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.015386 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d2df58-6860-403f-acb9-1b28d74a7791-istio-podinfo\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.042514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.028330 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-certs\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.042514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.028696 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-socket\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.042514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.029526 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d2df58-6860-403f-acb9-1b28d74a7791-istiod-ca-cert\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.042514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.029805 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-credential-socket\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.042514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.029926 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-data\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.053211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.043768 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d2df58-6860-403f-acb9-1b28d74a7791-istio-podinfo\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.053515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.050804 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-envoy\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.055291 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.055211 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqq7\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-kube-api-access-jfqq7\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.062110 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.062066 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-istio-token\") pod \"e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.066516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.066472 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:57.098467 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.098174 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:57.229536 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.225844 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:57.460117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.459626 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr"] Oct 31 01:16:57.460898 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:16:57.460869 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d2df58_6860_403f_acb9_1b28d74a7791.slice/crio-e20ecf86590d7ee6c707f717be3bff5384ce6a110b0cd8537f974c0fa96fd96d WatchSource:0}: Error finding container e20ecf86590d7ee6c707f717be3bff5384ce6a110b0cd8537f974c0fa96fd96d: Status 404 returned error can't find the container with id e20ecf86590d7ee6c707f717be3bff5384ce6a110b0cd8537f974c0fa96fd96d Oct 31 01:16:57.464056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.464022 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:16:57.464136 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.464100 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:16:57.464136 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:57.464132 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:16:58.129925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:58.129877 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" event={"ID":"20d2df58-6860-403f-acb9-1b28d74a7791","Type":"ContainerStarted","Data":"d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f"} Oct 31 01:16:58.130422 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:58.129967 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" event={"ID":"20d2df58-6860-403f-acb9-1b28d74a7791","Type":"ContainerStarted","Data":"e20ecf86590d7ee6c707f717be3bff5384ce6a110b0cd8537f974c0fa96fd96d"} Oct 31 01:16:58.184498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:58.184416 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" podStartSLOduration=2.184400605 podStartE2EDuration="2.184400605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:58.180444235 +0000 UTC m=+3435.878558003" watchObservedRunningTime="2026-10-31 01:16:58.184400605 +0000 UTC m=+3435.882514369" Oct 31 01:16:58.735488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:58.735027 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:59.099652 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:59.099606 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:59.105933 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:59.105904 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:59.136900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:59.136862 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:16:59.137904 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:16:59.137884 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:17:00.661086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:00.660973 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:00.853709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:00.853676 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:01.016722 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:01.016581 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:02.500268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.500231 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:02.618676 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.618640 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk"] Oct 31 01:17:02.623706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.623678 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:02.654522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.654489 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8d251f\"" Oct 31 01:17:02.654866 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.654850 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8d251f-dockercfg-wkhjv\"" Oct 31 01:17:02.725474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.719102 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77abf14-6901-4fee-94a6-fc5f6496a2de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk\" (UID: \"e77abf14-6901-4fee-94a6-fc5f6496a2de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:02.725474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.719331 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdqc\" (UniqueName: \"kubernetes.io/projected/e77abf14-6901-4fee-94a6-fc5f6496a2de-kube-api-access-zkdqc\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk\" (UID: \"e77abf14-6901-4fee-94a6-fc5f6496a2de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:02.729199 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.729162 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk"] Oct 31 01:17:02.823664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.823619 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdqc\" (UniqueName: \"kubernetes.io/projected/e77abf14-6901-4fee-94a6-fc5f6496a2de-kube-api-access-zkdqc\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk\" (UID: \"e77abf14-6901-4fee-94a6-fc5f6496a2de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:02.829402 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.829372 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77abf14-6901-4fee-94a6-fc5f6496a2de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk\" (UID: \"e77abf14-6901-4fee-94a6-fc5f6496a2de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:02.829696 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.829674 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77abf14-6901-4fee-94a6-fc5f6496a2de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk\" (UID: \"e77abf14-6901-4fee-94a6-fc5f6496a2de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:02.872062 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.872022 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:02.874006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.873982 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdqc\" (UniqueName: \"kubernetes.io/projected/e77abf14-6901-4fee-94a6-fc5f6496a2de-kube-api-access-zkdqc\") pod \"payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk\" (UID: \"e77abf14-6901-4fee-94a6-fc5f6496a2de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:02.939605 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:02.939567 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:03.052765 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:03.052728 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:03.183651 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:17:03.183566 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:17:03.248675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:03.248644 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk"] Oct 31 01:17:03.250496 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:17:03.250418 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77abf14_6901_4fee_94a6_fc5f6496a2de.slice/crio-644ebe58b0434e5308b9031e7da7ebc620eff45c7b43a447f08b7c8c7400378d WatchSource:0}: Error finding container 644ebe58b0434e5308b9031e7da7ebc620eff45c7b43a447f08b7c8c7400378d: Status 404 returned error can't find the container with id 644ebe58b0434e5308b9031e7da7ebc620eff45c7b43a447f08b7c8c7400378d Oct 31 01:17:04.174498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:04.174439 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" event={"ID":"e77abf14-6901-4fee-94a6-fc5f6496a2de","Type":"ContainerStarted","Data":"5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460"} Oct 31 01:17:04.175080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:04.174526 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" event={"ID":"e77abf14-6901-4fee-94a6-fc5f6496a2de","Type":"ContainerStarted","Data":"644ebe58b0434e5308b9031e7da7ebc620eff45c7b43a447f08b7c8c7400378d"} Oct 31 01:17:04.175080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:04.174699 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:04.214037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:04.213967 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" podStartSLOduration=2.213948376 podStartE2EDuration="2.213948376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:04.210041353 +0000 UTC m=+3441.908155130" watchObservedRunningTime="2026-10-31 01:17:04.213948376 +0000 UTC m=+3441.912062144" Oct 31 01:17:04.787935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:04.787325 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:05.933554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:05.933499 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:06.451765 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:06.451731 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:07.973785 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:07.973756 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:09.635692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:09.635652 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:11.330979 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:11.330735 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:12.877236 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:12.877189 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:14.515678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:14.515637 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:15.208315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:15.208284 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:17:16.322607 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:16.322550 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:17.750718 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:17.750680 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:17.909620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:17.909585 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:18.058133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:18.058046 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:18.396540 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:18.396501 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:18.554518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:18.554431 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:18.714941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:18.714862 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:19.033538 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:19.033402 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:20.091355 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:20.091318 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:20.229711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:20.229664 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:20.380957 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:20.380865 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:20.609475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:20.608596 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:22.778081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:22.778042 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:24.305569 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:24.305537 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:25.606727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:25.606691 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:25.751858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:25.751825 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:25.967548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:25.967420 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:26.130312 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:26.130271 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:27.893240 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:27.893203 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:29.971279 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:29.971225 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:31.510631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:31.510587 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:33.152196 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:33.152151 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:33.183801 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:17:33.183760 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:17:33.296352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:33.296315 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:35.271863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:35.269596 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:36.862119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:36.862082 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:38.408511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:38.408473 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:39.995398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:39.995356 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:41.523022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:41.522987 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:41.688504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:41.688469 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:41.833881 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:41.833845 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:43.211480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:43.211410 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:44.903747 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:44.903703 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:46.703137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:46.703096 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:48.344148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:48.344108 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:49.911176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:49.911144 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:50.055274 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:50.055240 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:51.616899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:51.616860 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:53.058157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:53.058122 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:53.303044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:53.302800 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:54.933552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:54.933511 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:56.278761 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:56.278714 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:56.442756 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:56.441546 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:57.386364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:57.386329 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:59.016292 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:17:59.016254 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:00.582145 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.582105 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:00.624867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.624825 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg"] Oct 31 01:18:00.631893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.631867 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.644381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.644359 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ada0c1-openshift-default-dockercfg-7hrph\"" Oct 31 01:18:00.661649 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.661622 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg"] Oct 31 01:18:00.684683 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.684643 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-data\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.684683 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.684695 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-certs\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.684991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.684826 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15d9e29-9873-4e96-bb76-d281d1a952ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.684991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.684903 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-token\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.684991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.684946 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-credential-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.685160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.685005 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-envoy\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.685160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.685040 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.685160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.685098 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-podinfo\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.685325 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.685219 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lpx\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-kube-api-access-x2lpx\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786198 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-podinfo\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786352 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lpx\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-kube-api-access-x2lpx\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786403 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-data\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786433 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-certs\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786548 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15d9e29-9873-4e96-bb76-d281d1a952ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786687 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786621 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-token\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786687 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786668 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-credential-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786790 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786746 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-envoy\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786886 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786862 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-data\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786883 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-certs\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.786950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.786862 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.787133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.787111 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-credential-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.787133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.787127 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-socket\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.787344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.787320 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15d9e29-9873-4e96-bb76-d281d1a952ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.789087 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.789063 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-envoy\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.789230 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.789211 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-podinfo\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.807367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.807337 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-token\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.807611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.807593 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lpx\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-kube-api-access-x2lpx\") pod \"e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:00.946007 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:00.945917 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:01.265326 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:01.265292 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg"] Oct 31 01:18:01.266855 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:18:01.266829 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15d9e29_9873_4e96_bb76_d281d1a952ff.slice/crio-7810b8cb199820d78f45664f1669afc8903adae560456adde1af97cf2d05e821 WatchSource:0}: Error finding container 7810b8cb199820d78f45664f1669afc8903adae560456adde1af97cf2d05e821: Status 404 returned error can't find the container with id 7810b8cb199820d78f45664f1669afc8903adae560456adde1af97cf2d05e821 Oct 31 01:18:01.269752 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:01.269717 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:01.269837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:01.269807 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:01.269882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:01.269836 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:01.760252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:01.760204 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" event={"ID":"b15d9e29-9873-4e96-bb76-d281d1a952ff","Type":"ContainerStarted","Data":"f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76"} Oct 31 01:18:01.760815 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:01.760301 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" event={"ID":"b15d9e29-9873-4e96-bb76-d281d1a952ff","Type":"ContainerStarted","Data":"7810b8cb199820d78f45664f1669afc8903adae560456adde1af97cf2d05e821"} Oct 31 01:18:01.824472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:01.824101 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" podStartSLOduration=1.824079737 podStartE2EDuration="1.824079737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:01.810383525 +0000 UTC m=+3499.508497305" watchObservedRunningTime="2026-10-31 01:18:01.824079737 +0000 UTC m=+3499.522193500" Oct 31 01:18:01.946377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:01.946340 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:02.322477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:02.320205 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:02.948874 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:02.948839 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 31 01:18:02.949305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:02.948905 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 31 01:18:03.184268 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:18:03.184228 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:18:03.946247 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:03.946212 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 31 01:18:03.946435 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:03.946268 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 31 01:18:04.613574 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:04.612789 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:04.946681 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:04.946594 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 31 01:18:04.946681 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:04.946654 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 31 01:18:05.946733 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:05.946692 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 31 01:18:05.947157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:05.946754 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 31 01:18:06.271530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:06.270752 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:06.946759 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:06.946725 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 31 01:18:06.947182 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:06.946783 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 31 01:18:07.889384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:07.889347 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:07.951653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:07.951619 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:07.952105 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:07.952080 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:07.952951 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:07.952929 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:18:09.252172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.252139 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq"] Oct 31 01:18:09.253926 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.252396 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" podUID="c774cdf8-8e12-4b28-87f2-713de41db619" containerName="maas-api" containerID="cri-o://82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4" gracePeriod=30 Oct 31 01:18:09.348694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.348647 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:09.401205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.401163 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq"] Oct 31 01:18:09.401490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.401436 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" podUID="1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66" containerName="payload-processing" containerID="cri-o://70c94eceeefb08175768bf9b5b1f3b1ce3a127878a59508c49e7690c5934a4b7" gracePeriod=30 Oct 31 01:18:09.520617 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.519213 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:09.604537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.604510 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:18:09.696091 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.696008 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:09.854712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.854652 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls\") pod \"c774cdf8-8e12-4b28-87f2-713de41db619\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " Oct 31 01:18:09.854948 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.854750 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c774cdf8-8e12-4b28-87f2-713de41db619-kube-api-access-m9c8s\" (UniqueName: \"kubernetes.io/projected/c774cdf8-8e12-4b28-87f2-713de41db619-kube-api-access-m9c8s\") pod \"c774cdf8-8e12-4b28-87f2-713de41db619\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " Oct 31 01:18:09.854948 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.854808 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c774cdf8-8e12-4b28-87f2-713de41db619-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c774cdf8-8e12-4b28-87f2-713de41db619-openshift-service-ca\") pod \"c774cdf8-8e12-4b28-87f2-713de41db619\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " Oct 31 01:18:09.855088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.855043 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-metrics-certs\") pod \"c774cdf8-8e12-4b28-87f2-713de41db619\" (UID: \"c774cdf8-8e12-4b28-87f2-713de41db619\") " Oct 31 01:18:09.859917 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.859783 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c774cdf8-8e12-4b28-87f2-713de41db619-openshift-service-ca" pod "c774cdf8-8e12-4b28-87f2-713de41db619" (UID: "c774cdf8-8e12-4b28-87f2-713de41db619"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:09.860073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.860002 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-metrics-certs" pod "c774cdf8-8e12-4b28-87f2-713de41db619" (UID: "c774cdf8-8e12-4b28-87f2-713de41db619"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:09.862824 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.862777 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c774cdf8-8e12-4b28-87f2-713de41db619-kube-api-access-m9c8s" pod "c774cdf8-8e12-4b28-87f2-713de41db619" (UID: "c774cdf8-8e12-4b28-87f2-713de41db619"). InnerVolumeSpecName "kube-api-access-m9c8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:09.869758 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.869352 5978 generic.go:356] "Generic (PLEG): container finished" podID="1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66" containerID="70c94eceeefb08175768bf9b5b1f3b1ce3a127878a59508c49e7690c5934a4b7" exitCode=0 Oct 31 01:18:09.869758 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.869660 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" event={"ID":"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66","Type":"ContainerDied","Data":"70c94eceeefb08175768bf9b5b1f3b1ce3a127878a59508c49e7690c5934a4b7"} Oct 31 01:18:09.879548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.876910 5978 generic.go:356] "Generic (PLEG): container finished" podID="c774cdf8-8e12-4b28-87f2-713de41db619" containerID="82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4" exitCode=0 Oct 31 01:18:09.879548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.878782 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" Oct 31 01:18:09.879548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.879310 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" event={"ID":"c774cdf8-8e12-4b28-87f2-713de41db619","Type":"ContainerDied","Data":"82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4"} Oct 31 01:18:09.879548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.879368 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq" event={"ID":"c774cdf8-8e12-4b28-87f2-713de41db619","Type":"ContainerDied","Data":"9ce74997227c8dda72a53407f89e67a23139473e86363e9effd47d9917517712"} Oct 31 01:18:09.879548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.879397 5978 scope.go:122] "RemoveContainer" containerID="82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4" Oct 31 01:18:09.879548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.879427 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls" pod "c774cdf8-8e12-4b28-87f2-713de41db619" (UID: "c774cdf8-8e12-4b28-87f2-713de41db619"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:09.893921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.893141 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:18:09.905201 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.905173 5978 scope.go:122] "RemoveContainer" containerID="82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4" Oct 31 01:18:09.905644 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:18:09.905617 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4\": container with ID starting with 82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4 not found: ID does not exist" containerID="82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4" Oct 31 01:18:09.905755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.905654 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4"} err="failed to get container status \"82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4\": rpc error: code = NotFound desc = could not find container \"82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4\": container with ID starting with 82f89159414cdba32f602712641b0fddfbcce6700834f3e7191e4a89435dc5d4 not found: ID does not exist" Oct 31 01:18:09.920704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.920670 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:09.958206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.958152 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-plugins-config-volume\") pod \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\" (UID: \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\") " Oct 31 01:18:09.958805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.958765 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-kube-api-access-rvkxr\" (UniqueName: \"kubernetes.io/projected/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-kube-api-access-rvkxr\") pod \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\" (UID: \"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66\") " Oct 31 01:18:09.959924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.959898 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-plugins-config-volume" pod "1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66" (UID: "1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:09.961773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.961745 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:09.961773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.961775 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9c8s\" (UniqueName: \"kubernetes.io/projected/c774cdf8-8e12-4b28-87f2-713de41db619-kube-api-access-m9c8s\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:09.961964 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.961793 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c774cdf8-8e12-4b28-87f2-713de41db619-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:09.961964 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.961808 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c774cdf8-8e12-4b28-87f2-713de41db619-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:09.963238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:09.963208 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-kube-api-access-rvkxr" pod "1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66" (UID: "1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66"). InnerVolumeSpecName "kube-api-access-rvkxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:10.063060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.063028 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:10.063060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.063060 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvkxr\" (UniqueName: \"kubernetes.io/projected/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66-kube-api-access-rvkxr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:10.066218 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.066164 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:10.212611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.212570 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:10.237816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.237782 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq"] Oct 31 01:18:10.244683 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.244647 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-33d44f-5b87584f4c-lknsq"] Oct 31 01:18:10.427534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.427501 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c774cdf8-8e12-4b28-87f2-713de41db619" path="/var/lib/kubelet/pods/c774cdf8-8e12-4b28-87f2-713de41db619/volumes" Oct 31 01:18:10.647763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.647727 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:10.887263 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.887221 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" event={"ID":"1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66","Type":"ContainerDied","Data":"7907623c4ae5117d15cd5adb1e068da72e1e73275c7a090a6f748032c98b314c"} Oct 31 01:18:10.887263 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.887252 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq" Oct 31 01:18:10.887580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.887307 5978 scope.go:122] "RemoveContainer" containerID="70c94eceeefb08175768bf9b5b1f3b1ce3a127878a59508c49e7690c5934a4b7" Oct 31 01:18:10.935827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.935782 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq"] Oct 31 01:18:10.945103 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:10.945031 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-33d44f-694fc9d78d-wc2mq"] Oct 31 01:18:11.991796 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:11.978862 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:12.427209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:12.427162 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66" path="/var/lib/kubelet/pods/1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66/volumes" Oct 31 01:18:13.304373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:13.304329 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:13.446092 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:13.446053 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:13.658846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:13.658798 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:13.824655 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:13.824613 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:15.901164 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:15.900983 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:16.974816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:16.974777 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:17.113404 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:17.113366 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:17.335386 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:17.335348 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:17.873345 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:17.873302 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:19.328029 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:19.327994 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg"] Oct 31 01:18:19.328551 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:19.328256 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" containerID="cri-o://3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a" gracePeriod=30 Oct 31 01:18:19.876226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:19.876187 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:21.510654 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:21.510616 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:22.105592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:22.105554 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl"] Oct 31 01:18:22.105858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:22.105800 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" containerID="cri-o://46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677" gracePeriod=30 Oct 31 01:18:23.097339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:23.097296 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:24.172528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.172426 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vd2tg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:18:24.172528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.172513 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:18:24.705149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.705119 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 01:18:24.716667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.716570 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:24.845900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.845853 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-data\") pod \"66a8a644-19f4-4af9-84bb-964324d42272\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " Oct 31 01:18:24.845900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.845916 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-certs\") pod \"66a8a644-19f4-4af9-84bb-964324d42272\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " Oct 31 01:18:24.846187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.845976 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-socket\") pod \"66a8a644-19f4-4af9-84bb-964324d42272\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " Oct 31 01:18:24.846187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.846006 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-envoy\") pod \"66a8a644-19f4-4af9-84bb-964324d42272\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " Oct 31 01:18:24.846187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.846025 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-istio-token\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-istio-token\") pod \"66a8a644-19f4-4af9-84bb-964324d42272\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " Oct 31 01:18:24.846187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.846053 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a8a644-19f4-4af9-84bb-964324d42272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8a644-19f4-4af9-84bb-964324d42272-istiod-ca-cert\") pod \"66a8a644-19f4-4af9-84bb-964324d42272\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " Oct 31 01:18:24.846187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.846137 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-credential-socket\") pod \"66a8a644-19f4-4af9-84bb-964324d42272\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " Oct 31 01:18:24.846187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.846169 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-kube-api-access-tqdhf\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-kube-api-access-tqdhf\") pod \"66a8a644-19f4-4af9-84bb-964324d42272\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " Oct 31 01:18:24.846535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.846206 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66a8a644-19f4-4af9-84bb-964324d42272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8a644-19f4-4af9-84bb-964324d42272-istio-podinfo\") pod \"66a8a644-19f4-4af9-84bb-964324d42272\" (UID: \"66a8a644-19f4-4af9-84bb-964324d42272\") " Oct 31 01:18:24.849165 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.849130 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-credential-socket" pod "66a8a644-19f4-4af9-84bb-964324d42272" (UID: "66a8a644-19f4-4af9-84bb-964324d42272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:24.849305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.849160 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-data" pod "66a8a644-19f4-4af9-84bb-964324d42272" (UID: "66a8a644-19f4-4af9-84bb-964324d42272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:24.849682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.849663 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-certs" pod "66a8a644-19f4-4af9-84bb-964324d42272" (UID: "66a8a644-19f4-4af9-84bb-964324d42272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:24.850259 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.850170 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a8a644-19f4-4af9-84bb-964324d42272-istiod-ca-cert" pod "66a8a644-19f4-4af9-84bb-964324d42272" (UID: "66a8a644-19f4-4af9-84bb-964324d42272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:24.850383 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.850262 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-socket" pod "66a8a644-19f4-4af9-84bb-964324d42272" (UID: "66a8a644-19f4-4af9-84bb-964324d42272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:24.851446 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.851422 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-istio-token" pod "66a8a644-19f4-4af9-84bb-964324d42272" (UID: "66a8a644-19f4-4af9-84bb-964324d42272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:24.851873 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.851844 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66a8a644-19f4-4af9-84bb-964324d42272-istio-podinfo" pod "66a8a644-19f4-4af9-84bb-964324d42272" (UID: "66a8a644-19f4-4af9-84bb-964324d42272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:18:24.852400 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.852381 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-envoy" pod "66a8a644-19f4-4af9-84bb-964324d42272" (UID: "66a8a644-19f4-4af9-84bb-964324d42272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:24.852895 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.852876 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-kube-api-access-tqdhf" pod "66a8a644-19f4-4af9-84bb-964324d42272" (UID: "66a8a644-19f4-4af9-84bb-964324d42272"). InnerVolumeSpecName "kube-api-access-tqdhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:24.948385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.948349 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:24.948385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.948378 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:24.948385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.948388 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:24.948385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.948397 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:24.948689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.948407 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:24.948689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.948415 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8a644-19f4-4af9-84bb-964324d42272-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:24.948689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.948423 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8a644-19f4-4af9-84bb-964324d42272-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:24.948689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.948433 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqdhf\" (UniqueName: \"kubernetes.io/projected/66a8a644-19f4-4af9-84bb-964324d42272-kube-api-access-tqdhf\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:24.948689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:24.948444 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8a644-19f4-4af9-84bb-964324d42272-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:25.025510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:25.025409 5978 generic.go:356] "Generic (PLEG): container finished" podID="66a8a644-19f4-4af9-84bb-964324d42272" containerID="3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a" exitCode=0 Oct 31 01:18:25.025510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:25.025495 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" Oct 31 01:18:25.025510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:25.025495 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" event={"ID":"66a8a644-19f4-4af9-84bb-964324d42272","Type":"ContainerDied","Data":"3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a"} Oct 31 01:18:25.025822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:25.025563 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg" event={"ID":"66a8a644-19f4-4af9-84bb-964324d42272","Type":"ContainerDied","Data":"af14ad0240c38bab33b7eadb6c68f2f0a1ecf17c30fb1f9d60a66b82b31b2df1"} Oct 31 01:18:25.025822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:25.025594 5978 scope.go:122] "RemoveContainer" containerID="3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a" Oct 31 01:18:25.038876 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:25.038852 5978 scope.go:122] "RemoveContainer" containerID="3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a" Oct 31 01:18:25.040361 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:18:25.040288 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a\": container with ID starting with 3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a not found: ID does not exist" containerID="3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a" Oct 31 01:18:25.040361 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:25.040326 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a"} err="failed to get container status \"3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a\": rpc error: code = NotFound desc = could not find container \"3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a\": container with ID starting with 3af2dd2b6a9ee9777f91bbcd3bbb37a60336bb997b28d35da61b880f09809a4a not found: ID does not exist" Oct 31 01:18:25.079119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:25.079081 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg"] Oct 31 01:18:25.082372 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:25.082343 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vd2tg"] Oct 31 01:18:26.439949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:26.439908 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a8a644-19f4-4af9-84bb-964324d42272" path="/var/lib/kubelet/pods/66a8a644-19f4-4af9-84bb-964324d42272/volumes" Oct 31 01:18:26.475394 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:26.475350 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:27.386975 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.386913 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:18:27.484117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484070 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-socket\") pod \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " Oct 31 01:18:27.484117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484124 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-credential-socket\") pod \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484213 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istiod-ca-cert\") pod \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484246 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-kube-api-access-j4xfn\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-kube-api-access-j4xfn\") pod \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484301 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-certs\") pod \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484342 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-envoy\") pod \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484383 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-token\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-token\") pod \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484541 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-data\") pod \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484583 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-podinfo\") pod \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\" (UID: \"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6\") " Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484634 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-credential-socket" pod "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" (UID: "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484690 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-certs" pod "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" (UID: "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.484913 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-data" pod "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" (UID: "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.485068 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-socket" pod "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" (UID: "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.485123 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istiod-ca-cert" pod "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" (UID: "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.485325 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.485346 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.485360 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.485374 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:27.485592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.485386 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:27.487488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.487428 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-podinfo" pod "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" (UID: "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:18:27.487610 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.487549 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-kube-api-access-j4xfn" pod "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" (UID: "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6"). InnerVolumeSpecName "kube-api-access-j4xfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:27.487684 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.487669 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-envoy" pod "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" (UID: "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:27.487940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.487916 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-token" pod "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" (UID: "bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:27.587207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.587159 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xfn\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-kube-api-access-j4xfn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:27.587207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.587199 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:27.587207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.587214 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:27.587520 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.587225 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:27.675374 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.675280 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk"] Oct 31 01:18:27.923984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:27.923946 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:28.052803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.052717 5978 generic.go:356] "Generic (PLEG): container finished" podID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerID="46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677" exitCode=0 Oct 31 01:18:28.052803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.052770 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" Oct 31 01:18:28.053042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.052799 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" event={"ID":"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6","Type":"ContainerDied","Data":"46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677"} Oct 31 01:18:28.053042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.052859 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl" event={"ID":"bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6","Type":"ContainerDied","Data":"16ca5fecdf8a94678b6c9579936b6451eb25f0cdea468f19fd4a0e496e97fa22"} Oct 31 01:18:28.053042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.052883 5978 scope.go:122] "RemoveContainer" containerID="46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677" Oct 31 01:18:28.065315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.065292 5978 scope.go:122] "RemoveContainer" containerID="46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677" Oct 31 01:18:28.065659 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:18:28.065629 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677\": container with ID starting with 46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677 not found: ID does not exist" containerID="46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677" Oct 31 01:18:28.065763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.065671 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677"} err="failed to get container status \"46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677\": rpc error: code = NotFound desc = could not find container \"46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677\": container with ID starting with 46e2c2dec4492ba232ac491dab8926c019c54eae04b1b999d03bc77552c3e677 not found: ID does not exist" Oct 31 01:18:28.080411 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.080378 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:28.119690 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.119655 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl"] Oct 31 01:18:28.133375 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.133339 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-498580-openshift-default-b66db7999-l4rtl"] Oct 31 01:18:28.251508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.251476 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:28.476472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.475581 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" path="/var/lib/kubelet/pods/bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6/volumes" Oct 31 01:18:28.793376 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.793258 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:28.938913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:28.938879 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:30.142477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:30.142232 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:31.831464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:31.831379 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:32.675983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:32.675928 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" podUID="e77abf14-6901-4fee-94a6-fc5f6496a2de" containerName="payload-pre-processing" containerID="cri-o://5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460" gracePeriod=25 Oct 31 01:18:33.030046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.029683 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:18:33.096750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.096717 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77abf14-6901-4fee-94a6-fc5f6496a2de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77abf14-6901-4fee-94a6-fc5f6496a2de-plugins-config-volume\") pod \"e77abf14-6901-4fee-94a6-fc5f6496a2de\" (UID: \"e77abf14-6901-4fee-94a6-fc5f6496a2de\") " Oct 31 01:18:33.096998 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.096977 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77abf14-6901-4fee-94a6-fc5f6496a2de-kube-api-access-zkdqc\" (UniqueName: \"kubernetes.io/projected/e77abf14-6901-4fee-94a6-fc5f6496a2de-kube-api-access-zkdqc\") pod \"e77abf14-6901-4fee-94a6-fc5f6496a2de\" (UID: \"e77abf14-6901-4fee-94a6-fc5f6496a2de\") " Oct 31 01:18:33.097191 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.097169 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77abf14-6901-4fee-94a6-fc5f6496a2de-plugins-config-volume" pod "e77abf14-6901-4fee-94a6-fc5f6496a2de" (UID: "e77abf14-6901-4fee-94a6-fc5f6496a2de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:33.097525 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.097509 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77abf14-6901-4fee-94a6-fc5f6496a2de-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:33.099307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.099278 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77abf14-6901-4fee-94a6-fc5f6496a2de-kube-api-access-zkdqc" pod "e77abf14-6901-4fee-94a6-fc5f6496a2de" (UID: "e77abf14-6901-4fee-94a6-fc5f6496a2de"). InnerVolumeSpecName "kube-api-access-zkdqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:33.183208 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:18:33.183166 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:18:33.197998 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.197959 5978 generic.go:356] "Generic (PLEG): container finished" podID="e77abf14-6901-4fee-94a6-fc5f6496a2de" containerID="5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460" exitCode=0 Oct 31 01:18:33.198179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.198007 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" Oct 31 01:18:33.198179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.198041 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" event={"ID":"e77abf14-6901-4fee-94a6-fc5f6496a2de","Type":"ContainerDied","Data":"5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460"} Oct 31 01:18:33.198179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.198105 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk" event={"ID":"e77abf14-6901-4fee-94a6-fc5f6496a2de","Type":"ContainerDied","Data":"644ebe58b0434e5308b9031e7da7ebc620eff45c7b43a447f08b7c8c7400378d"} Oct 31 01:18:33.198179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.198135 5978 scope.go:122] "RemoveContainer" containerID="5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460" Oct 31 01:18:33.198741 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.198717 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkdqc\" (UniqueName: \"kubernetes.io/projected/e77abf14-6901-4fee-94a6-fc5f6496a2de-kube-api-access-zkdqc\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:18:33.209492 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.209470 5978 scope.go:122] "RemoveContainer" containerID="5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460" Oct 31 01:18:33.209795 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:18:33.209773 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460\": container with ID starting with 5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460 not found: ID does not exist" containerID="5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460" Oct 31 01:18:33.209899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.209805 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460"} err="failed to get container status \"5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460\": rpc error: code = NotFound desc = could not find container \"5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460\": container with ID starting with 5a0984033e5aadecec8e794b190597f771ce013f0435d33acdf622e54e1c2460 not found: ID does not exist" Oct 31 01:18:33.255363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.255264 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk"] Oct 31 01:18:33.258631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.258605 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8d251f-69bcff9f-9fnjk"] Oct 31 01:18:33.588493 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:33.588408 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:34.443259 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:34.443219 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77abf14-6901-4fee-94a6-fc5f6496a2de" path="/var/lib/kubelet/pods/e77abf14-6901-4fee-94a6-fc5f6496a2de/volumes" Oct 31 01:18:35.208876 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:35.208824 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:36.854128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:36.854087 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:38.921127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:38.906116 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:40.269073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:40.269039 5978 generic.go:356] "Generic (PLEG): container finished" podID="20d2df58-6860-403f-acb9-1b28d74a7791" containerID="d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f" exitCode=0 Oct 31 01:18:40.269588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:40.269114 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" event={"ID":"20d2df58-6860-403f-acb9-1b28d74a7791","Type":"ContainerDied","Data":"d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f"} Oct 31 01:18:40.269700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:40.269679 5978 scope.go:122] "RemoveContainer" containerID="d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f" Oct 31 01:18:40.271588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:40.271547 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:40.271710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:40.271642 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:40.271710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:40.271695 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:40.528616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:40.523380 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:41.112641 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:41.112605 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:41.280172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:41.280139 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" event={"ID":"20d2df58-6860-403f-acb9-1b28d74a7791","Type":"ContainerStarted","Data":"b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9"} Oct 31 01:18:42.099951 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:42.099912 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:18:42.101360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:42.101335 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 01:18:42.101493 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:42.101400 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 01:18:42.239366 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:42.239330 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:43.062822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:43.050644 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:43.100497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:43.099711 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 01:18:43.100497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:43.099779 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 01:18:43.962576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:43.962531 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:44.099179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:44.099145 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 01:18:44.099653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:44.099199 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 01:18:44.137729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:44.137695 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:18:45.099813 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:45.099773 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 01:18:45.100231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:45.099842 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 01:18:45.964767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:45.964722 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:46.099951 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:46.099908 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 31 01:18:46.100468 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:46.099978 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 31 01:18:46.189703 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:46.189669 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:46.892771 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:46.892731 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:47.103785 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.103755 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:18:47.105046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.105026 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:18:47.345188 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.345153 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp"] Oct 31 01:18:47.346393 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346359 5978 cpu_manager.go:414] "containerMap: removing container" podUID="1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66" containerName="payload-processing" Oct 31 01:18:47.346393 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346387 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66" containerName="payload-processing" Oct 31 01:18:47.346651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346438 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c774cdf8-8e12-4b28-87f2-713de41db619" containerName="maas-api" Oct 31 01:18:47.346651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346447 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c774cdf8-8e12-4b28-87f2-713de41db619" containerName="maas-api" Oct 31 01:18:47.346651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346481 5978 cpu_manager.go:414] "containerMap: removing container" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" Oct 31 01:18:47.346651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346489 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" Oct 31 01:18:47.346651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346516 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e77abf14-6901-4fee-94a6-fc5f6496a2de" containerName="payload-pre-processing" Oct 31 01:18:47.346651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346524 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77abf14-6901-4fee-94a6-fc5f6496a2de" containerName="payload-pre-processing" Oct 31 01:18:47.346651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346547 5978 cpu_manager.go:414] "containerMap: removing container" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" Oct 31 01:18:47.346651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346555 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" Oct 31 01:18:47.347093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346712 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c774cdf8-8e12-4b28-87f2-713de41db619" containerName="maas-api" Oct 31 01:18:47.347093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346723 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a8a644-19f4-4af9-84bb-964324d42272" containerName="istio-proxy" Oct 31 01:18:47.347093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346741 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77abf14-6901-4fee-94a6-fc5f6496a2de" containerName="payload-pre-processing" Oct 31 01:18:47.347093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346751 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5879ea-e13f-4fd7-ad1e-8c5a0f4fc8c6" containerName="istio-proxy" Oct 31 01:18:47.347093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.346763 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6b4292-2d3c-4c9e-80bd-18cc48ed1f66" containerName="payload-processing" Oct 31 01:18:47.350904 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.350879 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.366987 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.366963 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-114c95-openshift-default-dockercfg-4tdr9\"" Oct 31 01:18:47.370002 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.369978 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp"] Oct 31 01:18:47.419310 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.419276 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:47.518360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.518318 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-certs\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.518764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.518734 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-credential-socket\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.518925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.518911 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52nlw\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-kube-api-access-52nlw\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.552348 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.552312 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-podinfo\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.552654 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.552632 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-envoy\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.552886 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.552870 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-token\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.553071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.553055 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-data\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.553199 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.553181 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bfd765-42c6-483e-ae9b-df8aff4cff55-istiod-ca-cert\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.553352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.553338 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-socket\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.583475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.580697 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:47.656409 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.656303 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-token\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.657126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.657097 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-data\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.657824 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.657803 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bfd765-42c6-483e-ae9b-df8aff4cff55-istiod-ca-cert\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.659024 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.659007 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-socket\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.659619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.659591 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-certs\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.659877 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.658940 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bfd765-42c6-483e-ae9b-df8aff4cff55-istiod-ca-cert\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.659958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.659387 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-socket\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.660066 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.657728 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-data\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.660497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.660186 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-certs\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.660642 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.660422 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-credential-socket\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.660757 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.660742 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-credential-socket\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.660894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.660881 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nlw\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-kube-api-access-52nlw\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.661397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.661379 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-podinfo\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.662032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.662015 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-envoy\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.671723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.671692 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-envoy\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.679032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.679003 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-podinfo\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.680999 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.680977 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-token\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.682154 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.682133 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52nlw\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-kube-api-access-52nlw\") pod \"e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:47.725013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.724979 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:47.971716 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:47.971625 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:48.297720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:48.297689 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp"] Oct 31 01:18:48.299871 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:18:48.299845 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bfd765_42c6_483e_ae9b_df8aff4cff55.slice/crio-994e294ce433a404d792fde209248d468bccdc4f985f73518097bfd378ec16c3 WatchSource:0}: Error finding container 994e294ce433a404d792fde209248d468bccdc4f985f73518097bfd378ec16c3: Status 404 returned error can't find the container with id 994e294ce433a404d792fde209248d468bccdc4f985f73518097bfd378ec16c3 Oct 31 01:18:48.303387 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:48.303347 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:48.303523 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:48.303478 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:48.303581 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:48.303524 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:18:48.360615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:48.360583 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" event={"ID":"03bfd765-42c6-483e-ae9b-df8aff4cff55","Type":"ContainerStarted","Data":"994e294ce433a404d792fde209248d468bccdc4f985f73518097bfd378ec16c3"} Oct 31 01:18:49.201146 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:49.201108 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:49.369900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:49.369861 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" event={"ID":"03bfd765-42c6-483e-ae9b-df8aff4cff55","Type":"ContainerStarted","Data":"dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede"} Oct 31 01:18:49.421305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:49.421255 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" podStartSLOduration=2.421242588 podStartE2EDuration="2.421242588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:49.420087286 +0000 UTC m=+3547.118201051" watchObservedRunningTime="2026-10-31 01:18:49.421242588 +0000 UTC m=+3547.119356353" Oct 31 01:18:49.972435 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:49.972396 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:49.978620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:49.978587 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:50.056588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:50.056548 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:50.215410 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:50.215377 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:50.377514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:50.377485 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:50.378494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:50.378446 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:18:50.967619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:50.967526 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:51.947160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:51.947125 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:52.103661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:52.103626 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:52.723508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:52.723448 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:54.076288 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.076250 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:54.160483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.160413 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds"] Oct 31 01:18:54.166901 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.166876 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:18:54.185693 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.185660 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-114c95\"" Oct 31 01:18:54.193551 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.193524 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-114c95-dockercfg-clsdf\"" Oct 31 01:18:54.198165 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.198134 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds"] Oct 31 01:18:54.224249 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.224211 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f"] Oct 31 01:18:54.234682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.234646 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:18:54.236210 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.236178 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:54.246087 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.246060 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f"] Oct 31 01:18:54.264125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.264054 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75jp2\" (UniqueName: \"kubernetes.io/projected/39e3ce88-34cf-4afb-8093-dc5809718072-kube-api-access-75jp2\") pod \"payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds\" (UID: \"39e3ce88-34cf-4afb-8093-dc5809718072\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:18:54.264322 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.264268 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e3ce88-34cf-4afb-8093-dc5809718072-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds\" (UID: \"39e3ce88-34cf-4afb-8093-dc5809718072\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:18:54.377561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.368173 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e3ce88-34cf-4afb-8093-dc5809718072-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds\" (UID: \"39e3ce88-34cf-4afb-8093-dc5809718072\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:18:54.377561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.368390 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6xq\" (UniqueName: \"kubernetes.io/projected/0180a3e1-bb04-48bd-babd-10eb6eb78853-kube-api-access-ht6xq\") pod \"payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f\" (UID: \"0180a3e1-bb04-48bd-babd-10eb6eb78853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:18:54.377561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.368516 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a3e1-bb04-48bd-babd-10eb6eb78853-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f\" (UID: \"0180a3e1-bb04-48bd-babd-10eb6eb78853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:18:54.377561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.368582 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75jp2\" (UniqueName: \"kubernetes.io/projected/39e3ce88-34cf-4afb-8093-dc5809718072-kube-api-access-75jp2\") pod \"payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds\" (UID: \"39e3ce88-34cf-4afb-8093-dc5809718072\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:18:54.377561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.369890 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e3ce88-34cf-4afb-8093-dc5809718072-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds\" (UID: \"39e3ce88-34cf-4afb-8093-dc5809718072\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:18:54.390818 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.390742 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75jp2\" (UniqueName: \"kubernetes.io/projected/39e3ce88-34cf-4afb-8093-dc5809718072-kube-api-access-75jp2\") pod \"payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds\" (UID: \"39e3ce88-34cf-4afb-8093-dc5809718072\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:18:54.469596 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.469533 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6xq\" (UniqueName: \"kubernetes.io/projected/0180a3e1-bb04-48bd-babd-10eb6eb78853-kube-api-access-ht6xq\") pod \"payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f\" (UID: \"0180a3e1-bb04-48bd-babd-10eb6eb78853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:18:54.469889 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.469651 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a3e1-bb04-48bd-babd-10eb6eb78853-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f\" (UID: \"0180a3e1-bb04-48bd-babd-10eb6eb78853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:18:54.470518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.470490 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a3e1-bb04-48bd-babd-10eb6eb78853-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f\" (UID: \"0180a3e1-bb04-48bd-babd-10eb6eb78853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:18:54.482516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.482489 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:18:54.491226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.491191 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6xq\" (UniqueName: \"kubernetes.io/projected/0180a3e1-bb04-48bd-babd-10eb6eb78853-kube-api-access-ht6xq\") pod \"payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f\" (UID: \"0180a3e1-bb04-48bd-babd-10eb6eb78853\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:18:54.557890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.557283 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:18:54.592695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.592638 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:54.789362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.789211 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds"] Oct 31 01:18:54.792368 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:18:54.792340 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e3ce88_34cf_4afb_8093_dc5809718072.slice/crio-444f4c75f41acb19426b39ead3ce59ec74bb7b8f0b42878ac4be3b638618b426 WatchSource:0}: Error finding container 444f4c75f41acb19426b39ead3ce59ec74bb7b8f0b42878ac4be3b638618b426: Status 404 returned error can't find the container with id 444f4c75f41acb19426b39ead3ce59ec74bb7b8f0b42878ac4be3b638618b426 Oct 31 01:18:54.918072 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:54.918040 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f"] Oct 31 01:18:54.921527 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:18:54.921446 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0180a3e1_bb04_48bd_babd_10eb6eb78853.slice/crio-acc5e7baaade155f1e5119782a3a10f30d514ec1a3931b806eb70fe76c55f872 WatchSource:0}: Error finding container acc5e7baaade155f1e5119782a3a10f30d514ec1a3931b806eb70fe76c55f872: Status 404 returned error can't find the container with id acc5e7baaade155f1e5119782a3a10f30d514ec1a3931b806eb70fe76c55f872 Oct 31 01:18:55.430171 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:55.430135 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" event={"ID":"0180a3e1-bb04-48bd-babd-10eb6eb78853","Type":"ContainerStarted","Data":"537f90b4c7e426d1db41ebb1375a769563a21ddbdb830249a7b8bd01830cf2f9"} Oct 31 01:18:55.430653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:55.430191 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" event={"ID":"0180a3e1-bb04-48bd-babd-10eb6eb78853","Type":"ContainerStarted","Data":"acc5e7baaade155f1e5119782a3a10f30d514ec1a3931b806eb70fe76c55f872"} Oct 31 01:18:55.430653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:55.430219 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:18:55.432010 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:55.431983 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" event={"ID":"39e3ce88-34cf-4afb-8093-dc5809718072","Type":"ContainerStarted","Data":"e1032af286ba127378b54f5b8934c25abbc278329074dd293ddcbd62d7c527d6"} Oct 31 01:18:55.432158 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:55.432023 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" event={"ID":"39e3ce88-34cf-4afb-8093-dc5809718072","Type":"ContainerStarted","Data":"444f4c75f41acb19426b39ead3ce59ec74bb7b8f0b42878ac4be3b638618b426"} Oct 31 01:18:55.432158 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:55.432071 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:18:55.471829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:55.471771 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" podStartSLOduration=1.47175579 podStartE2EDuration="1.47175579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:55.470965863 +0000 UTC m=+3553.169079660" watchObservedRunningTime="2026-10-31 01:18:55.47175579 +0000 UTC m=+3553.169869629" Oct 31 01:18:55.509470 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:55.509412 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" podStartSLOduration=1.5093975309999998 podStartE2EDuration="1.509397531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:55.506058422 +0000 UTC m=+3553.204172189" watchObservedRunningTime="2026-10-31 01:18:55.509397531 +0000 UTC m=+3553.207511296" Oct 31 01:18:56.785043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:56.782628 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:58.414750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:18:58.413425 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:00.032447 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:00.018072 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:01.379890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:01.379855 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:01.884792 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:01.884753 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:03.182274 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:19:03.182231 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:19:03.705179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:03.705138 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:04.964333 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:04.964285 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:05.115018 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.114981 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:05.441522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.439242 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt"] Oct 31 01:19:05.441522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.439491 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" containerID="cri-o://3b536a3c52d10b7fb70a62ad0861231cd6ad0086a7076b46773cffe140818bc4" gracePeriod=30 Oct 31 01:19:05.501081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.501044 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc"] Oct 31 01:19:05.506846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.506818 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:05.522795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.522766 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8d251f\"" Oct 31 01:19:05.522947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.522798 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8d251f-dockercfg-zvrnb\"" Oct 31 01:19:05.529728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.529693 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc"] Oct 31 01:19:05.667685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.667650 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc\" (UID: \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:05.667874 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.667795 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjzqx\" (UniqueName: \"kubernetes.io/projected/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-kube-api-access-hjzqx\") pod \"payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc\" (UID: \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:05.769746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.769659 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjzqx\" (UniqueName: \"kubernetes.io/projected/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-kube-api-access-hjzqx\") pod \"payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc\" (UID: \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:05.769935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.769843 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc\" (UID: \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:05.770515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.770495 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc\" (UID: \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:05.796346 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.796319 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjzqx\" (UniqueName: \"kubernetes.io/projected/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-kube-api-access-hjzqx\") pod \"payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc\" (UID: \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:05.827507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:05.827473 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:06.132150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:06.132125 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc"] Oct 31 01:19:06.134257 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:19:06.134230 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0a8c9f_dfe2_4a7d_b4f7_bbc869ae5f6a.slice/crio-ff72b0e977e1eb4db0ddc57f8bd0eb852e819d24a55e1a74688ecba76f8e4d52 WatchSource:0}: Error finding container ff72b0e977e1eb4db0ddc57f8bd0eb852e819d24a55e1a74688ecba76f8e4d52: Status 404 returned error can't find the container with id ff72b0e977e1eb4db0ddc57f8bd0eb852e819d24a55e1a74688ecba76f8e4d52 Oct 31 01:19:06.286084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:06.286051 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:06.441407 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:06.441330 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:19:06.441407 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:06.441395 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:19:06.531328 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:06.531285 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" event={"ID":"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a","Type":"ContainerStarted","Data":"a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782"} Oct 31 01:19:06.531647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:06.531356 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" event={"ID":"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a","Type":"ContainerStarted","Data":"ff72b0e977e1eb4db0ddc57f8bd0eb852e819d24a55e1a74688ecba76f8e4d52"} Oct 31 01:19:06.531934 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:06.531893 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:06.572285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:06.571908 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" podStartSLOduration=1.57188884 podStartE2EDuration="1.57188884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:06.57039621 +0000 UTC m=+3564.268509978" watchObservedRunningTime="2026-10-31 01:19:06.57188884 +0000 UTC m=+3564.270002606" Oct 31 01:19:07.853443 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:07.853392 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:08.535571 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:08.535536 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:08.715560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:08.705051 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:08.902760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:08.902725 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:09.190521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:09.190414 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:10.248534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.248495 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:19:10.248948 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.248577 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:19:10.530203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.530115 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:10.561793 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.561752 5978 generic.go:356] "Generic (PLEG): container finished" podID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerID="3b536a3c52d10b7fb70a62ad0861231cd6ad0086a7076b46773cffe140818bc4" exitCode=0 Oct 31 01:19:10.561920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.561817 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" event={"ID":"163792a7-efc3-4a2b-96cf-a21b004b7f20","Type":"ContainerDied","Data":"3b536a3c52d10b7fb70a62ad0861231cd6ad0086a7076b46773cffe140818bc4"} Oct 31 01:19:10.561920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.561905 5978 scope.go:122] "RemoveContainer" containerID="f5a5ba0a3503f2068ce94b0b9ca758f3768a15652f60fb2e7c0829965c3fe17a" Oct 31 01:19:10.734198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.734169 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:19:10.771837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.771805 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-token\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-token\") pod \"163792a7-efc3-4a2b-96cf-a21b004b7f20\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " Oct 31 01:19:10.772014 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.771891 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-credential-socket\") pod \"163792a7-efc3-4a2b-96cf-a21b004b7f20\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " Oct 31 01:19:10.772014 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.771967 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-certs\") pod \"163792a7-efc3-4a2b-96cf-a21b004b7f20\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " Oct 31 01:19:10.772150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772011 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163792a7-efc3-4a2b-96cf-a21b004b7f20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163792a7-efc3-4a2b-96cf-a21b004b7f20-istiod-ca-cert\") pod \"163792a7-efc3-4a2b-96cf-a21b004b7f20\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " Oct 31 01:19:10.772150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772064 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-data\") pod \"163792a7-efc3-4a2b-96cf-a21b004b7f20\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " Oct 31 01:19:10.772267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772191 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-socket\") pod \"163792a7-efc3-4a2b-96cf-a21b004b7f20\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " Oct 31 01:19:10.772267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772220 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-podinfo\") pod \"163792a7-efc3-4a2b-96cf-a21b004b7f20\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " Oct 31 01:19:10.772267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772194 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-credential-socket" pod "163792a7-efc3-4a2b-96cf-a21b004b7f20" (UID: "163792a7-efc3-4a2b-96cf-a21b004b7f20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:10.772423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772276 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-certs" pod "163792a7-efc3-4a2b-96cf-a21b004b7f20" (UID: "163792a7-efc3-4a2b-96cf-a21b004b7f20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:10.772423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772384 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-data" pod "163792a7-efc3-4a2b-96cf-a21b004b7f20" (UID: "163792a7-efc3-4a2b-96cf-a21b004b7f20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:10.772423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772387 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-kube-api-access-2q6v7\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-kube-api-access-2q6v7\") pod \"163792a7-efc3-4a2b-96cf-a21b004b7f20\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " Oct 31 01:19:10.772614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772482 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-socket" pod "163792a7-efc3-4a2b-96cf-a21b004b7f20" (UID: "163792a7-efc3-4a2b-96cf-a21b004b7f20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:10.772614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772512 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-envoy\") pod \"163792a7-efc3-4a2b-96cf-a21b004b7f20\" (UID: \"163792a7-efc3-4a2b-96cf-a21b004b7f20\") " Oct 31 01:19:10.772750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.772513 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163792a7-efc3-4a2b-96cf-a21b004b7f20-istiod-ca-cert" pod "163792a7-efc3-4a2b-96cf-a21b004b7f20" (UID: "163792a7-efc3-4a2b-96cf-a21b004b7f20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:10.773104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.773081 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163792a7-efc3-4a2b-96cf-a21b004b7f20-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:10.773104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.773105 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:10.773279 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.773118 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:10.773279 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.773132 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:10.773279 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.773144 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:10.774589 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.774555 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-envoy" pod "163792a7-efc3-4a2b-96cf-a21b004b7f20" (UID: "163792a7-efc3-4a2b-96cf-a21b004b7f20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:10.774708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.774635 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-kube-api-access-2q6v7" pod "163792a7-efc3-4a2b-96cf-a21b004b7f20" (UID: "163792a7-efc3-4a2b-96cf-a21b004b7f20"). InnerVolumeSpecName "kube-api-access-2q6v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:10.774708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.774650 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-token" pod "163792a7-efc3-4a2b-96cf-a21b004b7f20" (UID: "163792a7-efc3-4a2b-96cf-a21b004b7f20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:10.774801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.774721 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-podinfo" pod "163792a7-efc3-4a2b-96cf-a21b004b7f20" (UID: "163792a7-efc3-4a2b-96cf-a21b004b7f20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:19:10.874907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.874868 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:10.874907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.874898 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q6v7\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-kube-api-access-2q6v7\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:10.874907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.874908 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:10.874907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:10.874918 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163792a7-efc3-4a2b-96cf-a21b004b7f20-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:11.571878 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:11.571833 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" event={"ID":"163792a7-efc3-4a2b-96cf-a21b004b7f20","Type":"ContainerDied","Data":"0db3340fbc500f98614b101a44cc9d8f4db255b34c99c0ca66558fd47dfa07d7"} Oct 31 01:19:11.572303 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:11.571913 5978 scope.go:122] "RemoveContainer" containerID="3b536a3c52d10b7fb70a62ad0861231cd6ad0086a7076b46773cffe140818bc4" Oct 31 01:19:11.572303 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:11.572045 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt" Oct 31 01:19:11.620548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:11.620504 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt"] Oct 31 01:19:11.629880 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:11.629844 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-99a5c2-openshift-default-6d6895cd99-4w8jt"] Oct 31 01:19:12.372676 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:12.372641 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:12.490091 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:12.490049 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" path="/var/lib/kubelet/pods/163792a7-efc3-4a2b-96cf-a21b004b7f20/volumes" Oct 31 01:19:12.671834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:12.671730 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:12.829464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:12.829413 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:14.110164 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:14.110120 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:15.713257 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:15.713223 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:15.853706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:15.853669 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:16.055966 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:16.055872 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:17.542650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:17.542616 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:19:17.607845 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:17.607807 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:19.759857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:19.759822 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:21.502180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:21.501978 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:23.208704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:23.208669 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:24.988924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:24.988330 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:27.020969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:27.020913 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:27.221261 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:27.221217 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:27.358905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:27.358873 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:28.612114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:28.612074 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:29.620231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.620198 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv"] Oct 31 01:19:29.620988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.620971 5978 cpu_manager.go:414] "containerMap: removing container" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" Oct 31 01:19:29.621033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.620989 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" Oct 31 01:19:29.621116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.621107 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" Oct 31 01:19:29.621166 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.621120 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" Oct 31 01:19:29.626125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.626099 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.639354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.639328 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4207cd-openshift-default-dockercfg-gsk62\"" Oct 31 01:19:29.654339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.654305 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv"] Oct 31 01:19:29.688242 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.688201 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b232e00-9245-48ae-a016-2c0320dbdb02-istiod-ca-cert\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.688445 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.688261 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-podinfo\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.688445 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.688397 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-certs\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.688445 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.688433 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-credential-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.688659 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.688494 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-token\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.688719 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.688672 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-data\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.688719 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.688711 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-envoy\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.688840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.688819 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znr9f\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-kube-api-access-znr9f\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.688887 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.688865 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.790517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790433 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.790714 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790658 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b232e00-9245-48ae-a016-2c0320dbdb02-istiod-ca-cert\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.790714 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790682 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-podinfo\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.790834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790724 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-certs\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.790834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790751 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-credential-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.790834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790792 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-token\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.790834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790817 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.791141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790901 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-data\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.791141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790935 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-envoy\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.791141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.790975 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znr9f\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-kube-api-access-znr9f\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.791141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.791079 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-certs\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.791377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.791218 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-credential-socket\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.791377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.791232 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-data\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.791554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.791503 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b232e00-9245-48ae-a016-2c0320dbdb02-istiod-ca-cert\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.793378 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.793355 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-envoy\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.793447 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.793375 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-podinfo\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.811755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.811719 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-token\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.811886 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.811789 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znr9f\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-kube-api-access-znr9f\") pod \"e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:29.941324 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:29.940948 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:30.109864 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:30.109816 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:30.264506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:30.264477 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv"] Oct 31 01:19:30.266392 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:19:30.266348 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b232e00_9245_48ae_a016_2c0320dbdb02.slice/crio-343c9ee01dfd937c8cb76803c5f32b90c6b2989abff586221fbe51a8644eef46 WatchSource:0}: Error finding container 343c9ee01dfd937c8cb76803c5f32b90c6b2989abff586221fbe51a8644eef46: Status 404 returned error can't find the container with id 343c9ee01dfd937c8cb76803c5f32b90c6b2989abff586221fbe51a8644eef46 Oct 31 01:19:30.269235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:30.269196 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:19:30.269328 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:30.269270 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:19:30.269328 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:30.269303 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:19:30.318445 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:19:30.318415 5978 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b232e00_9245_48ae_a016_2c0320dbdb02.slice/crio-23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:19:30.740829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:30.740796 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" event={"ID":"6b232e00-9245-48ae-a016-2c0320dbdb02","Type":"ContainerStarted","Data":"23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f"} Oct 31 01:19:30.741317 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:30.740849 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" event={"ID":"6b232e00-9245-48ae-a016-2c0320dbdb02","Type":"ContainerStarted","Data":"343c9ee01dfd937c8cb76803c5f32b90c6b2989abff586221fbe51a8644eef46"} Oct 31 01:19:30.803814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:30.803684 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" podStartSLOduration=1.8036599739999999 podStartE2EDuration="1.803659974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:30.790204641 +0000 UTC m=+3588.488318407" watchObservedRunningTime="2026-10-31 01:19:30.803659974 +0000 UTC m=+3588.501773740" Oct 31 01:19:30.946076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:30.946046 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:31.760832 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:31.760799 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:31.903415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:31.901105 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:31.948214 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:31.947953 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:19:31.948214 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:31.948006 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 01:19:32.947010 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:32.946974 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:19:32.947502 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:32.947032 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 01:19:33.182352 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:19:33.182312 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:19:33.525354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:33.525313 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:33.946051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:33.946008 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:19:33.946264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:33.946090 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 01:19:34.946651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:34.946613 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:19:34.947081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:34.946671 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 01:19:35.158874 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:35.158817 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:35.947034 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:35.946992 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:19:35.947505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:35.947058 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 01:19:36.839535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:36.839422 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:36.946349 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:36.946310 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 31 01:19:36.946558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:36.946371 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 31 01:19:37.951260 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:37.951221 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:37.952639 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:37.952616 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:37.954107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:37.954087 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:19:38.798986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:38.798950 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:39.344907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:39.344873 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:39.498526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:39.498490 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:39.647317 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:39.647223 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:39.997409 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:39.997321 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:41.307384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:41.307342 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:42.894366 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:42.894309 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:44.537681 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:44.537640 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:46.362556 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:46.362516 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:46.520983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:46.520937 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:46.664558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:46.664431 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:46.891697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:46.891658 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:48.425914 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:48.425871 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:50.151094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:50.151058 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:50.290695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:50.290658 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:50.653992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:50.653962 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:19:50.660796 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:50.660773 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:19:51.816277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:51.816228 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:53.034075 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.034041 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld"] Oct 31 01:19:53.034694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.034326 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" podUID="f01e023e-3b05-4ca5-a4e2-19c682dcc570" containerName="maas-api" containerID="cri-o://76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a" gracePeriod=30 Oct 31 01:19:53.133055 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.133019 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:53.367717 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.367692 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:19:53.501478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.498281 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01e023e-3b05-4ca5-a4e2-19c682dcc570-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e023e-3b05-4ca5-a4e2-19c682dcc570-openshift-service-ca\") pod \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " Oct 31 01:19:53.501478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.498372 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-maas-api-tls\") pod \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " Oct 31 01:19:53.501478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.498531 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01e023e-3b05-4ca5-a4e2-19c682dcc570-kube-api-access-kqc6x\" (UniqueName: \"kubernetes.io/projected/f01e023e-3b05-4ca5-a4e2-19c682dcc570-kube-api-access-kqc6x\") pod \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " Oct 31 01:19:53.501478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.498577 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-metrics-certs\") pod \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\" (UID: \"f01e023e-3b05-4ca5-a4e2-19c682dcc570\") " Oct 31 01:19:53.501478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.501060 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01e023e-3b05-4ca5-a4e2-19c682dcc570-openshift-service-ca" pod "f01e023e-3b05-4ca5-a4e2-19c682dcc570" (UID: "f01e023e-3b05-4ca5-a4e2-19c682dcc570"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:53.537236 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.531644 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-maas-api-tls" pod "f01e023e-3b05-4ca5-a4e2-19c682dcc570" (UID: "f01e023e-3b05-4ca5-a4e2-19c682dcc570"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:53.546547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.540709 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01e023e-3b05-4ca5-a4e2-19c682dcc570-kube-api-access-kqc6x" pod "f01e023e-3b05-4ca5-a4e2-19c682dcc570" (UID: "f01e023e-3b05-4ca5-a4e2-19c682dcc570"). InnerVolumeSpecName "kube-api-access-kqc6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:53.546547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.543417 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-metrics-certs" pod "f01e023e-3b05-4ca5-a4e2-19c682dcc570" (UID: "f01e023e-3b05-4ca5-a4e2-19c682dcc570"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:53.601553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.601400 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e023e-3b05-4ca5-a4e2-19c682dcc570-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:53.601553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.601448 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:53.601553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.601481 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqc6x\" (UniqueName: \"kubernetes.io/projected/f01e023e-3b05-4ca5-a4e2-19c682dcc570-kube-api-access-kqc6x\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:53.601553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.601495 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e023e-3b05-4ca5-a4e2-19c682dcc570-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:19:53.790627 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.790535 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:53.932330 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.932229 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:53.944813 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.944781 5978 generic.go:356] "Generic (PLEG): container finished" podID="f01e023e-3b05-4ca5-a4e2-19c682dcc570" containerID="76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a" exitCode=0 Oct 31 01:19:53.944980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.944823 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" event={"ID":"f01e023e-3b05-4ca5-a4e2-19c682dcc570","Type":"ContainerDied","Data":"76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a"} Oct 31 01:19:53.944980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.944838 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" Oct 31 01:19:53.944980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.944873 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld" event={"ID":"f01e023e-3b05-4ca5-a4e2-19c682dcc570","Type":"ContainerDied","Data":"1b3329c1b36f96c563c6773beefa8c77e690989ae45af13a7022b4a00db18617"} Oct 31 01:19:53.944980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.944898 5978 scope.go:122] "RemoveContainer" containerID="76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a" Oct 31 01:19:53.957349 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.957321 5978 scope.go:122] "RemoveContainer" containerID="76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a" Oct 31 01:19:53.957763 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:19:53.957726 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a\": container with ID starting with 76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a not found: ID does not exist" containerID="76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a" Oct 31 01:19:53.957891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.957769 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a"} err="failed to get container status \"76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a\": rpc error: code = NotFound desc = could not find container \"76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a\": container with ID starting with 76acfa78414822c303ef33a09f9a478f6d3ad7acada224d04c8ddb0eb1f7223a not found: ID does not exist" Oct 31 01:19:53.992557 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:53.992524 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld"] Oct 31 01:19:54.000137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:54.000098 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b8fd5a-867dc6f666-8hmld"] Oct 31 01:19:54.418127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:54.418090 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:54.426550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:54.426518 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01e023e-3b05-4ca5-a4e2-19c682dcc570" path="/var/lib/kubelet/pods/f01e023e-3b05-4ca5-a4e2-19c682dcc570/volumes" Oct 31 01:19:55.732148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:55.732116 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:57.818523 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:57.817676 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:58.011805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:58.011755 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:58.160197 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:58.160109 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:59.433809 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:19:59.433758 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:00.755353 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:00.755311 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:00.928207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:00.928172 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.123878 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:01.123833 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.275774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:01.275736 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.655202 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:01.655161 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:02.734600 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:02.734549 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:02.941574 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:02.941537 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:03.086081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:03.086042 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:03.291521 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:20:03.288961 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:20:03.298213 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:03.298169 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:03.472222 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:03.472091 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:03.957391 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:03.957358 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:05.312795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:05.312765 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:06.554951 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:06.554914 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx"] Oct 31 01:20:06.564645 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:06.564619 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp"] Oct 31 01:20:06.705521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:06.705482 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:06.899640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:06.899600 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:08.562185 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:08.562153 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:10.227889 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:10.227843 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:10.377028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:10.376993 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:10.523991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:10.523904 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:10.745692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:10.745644 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:12.100058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:12.100021 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:12.409978 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:12.409895 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj"] Oct 31 01:20:12.443508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:12.443466 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl"] Oct 31 01:20:12.599432 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:12.599391 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:12.740504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:12.740378 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:13.832511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:13.829096 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds"] Oct 31 01:20:13.856306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:13.856268 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f"] Oct 31 01:20:14.004249 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:14.004213 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:14.174051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:14.173958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:14.338840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:14.338800 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:16.279162 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:16.279110 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:17.410334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:17.410287 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" podUID="c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" containerName="payload-processing" containerID="cri-o://609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df" gracePeriod=25 Oct 31 01:20:17.443883 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:17.443842 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" podUID="f22077e0-eac8-45ef-92fd-ac6f597563fa" containerName="payload-pre-processing" containerID="cri-o://6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6" gracePeriod=25 Oct 31 01:20:17.544714 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:17.542732 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7"] Oct 31 01:20:17.544714 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:17.542981 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" podUID="52036be9-339e-4bb6-a5c1-30f41c943ff5" containerName="maas-api" containerID="cri-o://3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a" gracePeriod=30 Oct 31 01:20:17.643597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:17.643558 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:18.107985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.107951 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 01:20:18.115507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.115438 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:18.173281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.173243 5978 generic.go:356] "Generic (PLEG): container finished" podID="52036be9-339e-4bb6-a5c1-30f41c943ff5" containerID="3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a" exitCode=0 Oct 31 01:20:18.173508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.173492 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" event={"ID":"52036be9-339e-4bb6-a5c1-30f41c943ff5","Type":"ContainerDied","Data":"3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a"} Oct 31 01:20:18.173568 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.173536 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" event={"ID":"52036be9-339e-4bb6-a5c1-30f41c943ff5","Type":"ContainerDied","Data":"1a20a4cf361e4b751598d1f9559409182af04aebe12b1e19760bc692538612a0"} Oct 31 01:20:18.173568 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.173559 5978 scope.go:122] "RemoveContainer" containerID="3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a" Oct 31 01:20:18.173698 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.173688 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7" Oct 31 01:20:18.190935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.190901 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52036be9-339e-4bb6-a5c1-30f41c943ff5-kube-api-access-g7h5c\" (UniqueName: \"kubernetes.io/projected/52036be9-339e-4bb6-a5c1-30f41c943ff5-kube-api-access-g7h5c\") pod \"52036be9-339e-4bb6-a5c1-30f41c943ff5\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " Oct 31 01:20:18.191173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.191151 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-metrics-certs\") pod \"52036be9-339e-4bb6-a5c1-30f41c943ff5\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " Oct 31 01:20:18.191265 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.191245 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52036be9-339e-4bb6-a5c1-30f41c943ff5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036be9-339e-4bb6-a5c1-30f41c943ff5-openshift-service-ca\") pod \"52036be9-339e-4bb6-a5c1-30f41c943ff5\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " Oct 31 01:20:18.191324 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.191299 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls\") pod \"52036be9-339e-4bb6-a5c1-30f41c943ff5\" (UID: \"52036be9-339e-4bb6-a5c1-30f41c943ff5\") " Oct 31 01:20:18.192574 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.191994 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52036be9-339e-4bb6-a5c1-30f41c943ff5-openshift-service-ca" pod "52036be9-339e-4bb6-a5c1-30f41c943ff5" (UID: "52036be9-339e-4bb6-a5c1-30f41c943ff5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:18.192763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.192630 5978 scope.go:122] "RemoveContainer" containerID="3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a" Oct 31 01:20:18.193188 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:20:18.193111 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a\": container with ID starting with 3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a not found: ID does not exist" containerID="3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a" Oct 31 01:20:18.193188 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.193153 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a"} err="failed to get container status \"3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a\": rpc error: code = NotFound desc = could not find container \"3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a\": container with ID starting with 3a3e7658f807f9145171a1486042eecfcccb0dd3e9e4ba63f3e49955a5e7814a not found: ID does not exist" Oct 31 01:20:18.193715 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.193684 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-metrics-certs" pod "52036be9-339e-4bb6-a5c1-30f41c943ff5" (UID: "52036be9-339e-4bb6-a5c1-30f41c943ff5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:18.194329 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.194305 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52036be9-339e-4bb6-a5c1-30f41c943ff5-kube-api-access-g7h5c" pod "52036be9-339e-4bb6-a5c1-30f41c943ff5" (UID: "52036be9-339e-4bb6-a5c1-30f41c943ff5"). InnerVolumeSpecName "kube-api-access-g7h5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:18.194464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.194430 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls" pod "52036be9-339e-4bb6-a5c1-30f41c943ff5" (UID: "52036be9-339e-4bb6-a5c1-30f41c943ff5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:18.270178 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.270115 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:18.306515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.306093 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:18.306515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.306133 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52036be9-339e-4bb6-a5c1-30f41c943ff5-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:18.306515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.306150 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52036be9-339e-4bb6-a5c1-30f41c943ff5-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:18.306515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.306175 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7h5c\" (UniqueName: \"kubernetes.io/projected/52036be9-339e-4bb6-a5c1-30f41c943ff5-kube-api-access-g7h5c\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:18.517082 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.516429 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7"] Oct 31 01:20:18.539746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.539704 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3c90d8-6f87f66778-gsfn7"] Oct 31 01:20:18.788585 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.788487 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:18.835703 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.835609 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" podUID="39e3ce88-34cf-4afb-8093-dc5809718072" containerName="payload-processing" containerID="cri-o://e1032af286ba127378b54f5b8934c25abbc278329074dd293ddcbd62d7c527d6" gracePeriod=25 Oct 31 01:20:18.857154 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:18.857000 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" podUID="0180a3e1-bb04-48bd-babd-10eb6eb78853" containerName="payload-pre-processing" containerID="cri-o://537f90b4c7e426d1db41ebb1375a769563a21ddbdb830249a7b8bd01830cf2f9" gracePeriod=25 Oct 31 01:20:19.182987 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.182956 5978 generic.go:356] "Generic (PLEG): container finished" podID="39e3ce88-34cf-4afb-8093-dc5809718072" containerID="e1032af286ba127378b54f5b8934c25abbc278329074dd293ddcbd62d7c527d6" exitCode=0 Oct 31 01:20:19.183128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.183034 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" event={"ID":"39e3ce88-34cf-4afb-8093-dc5809718072","Type":"ContainerDied","Data":"e1032af286ba127378b54f5b8934c25abbc278329074dd293ddcbd62d7c527d6"} Oct 31 01:20:19.183128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.183102 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" event={"ID":"39e3ce88-34cf-4afb-8093-dc5809718072","Type":"ContainerDied","Data":"444f4c75f41acb19426b39ead3ce59ec74bb7b8f0b42878ac4be3b638618b426"} Oct 31 01:20:19.183255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.183129 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="444f4c75f41acb19426b39ead3ce59ec74bb7b8f0b42878ac4be3b638618b426" Oct 31 01:20:19.184689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.184663 5978 generic.go:356] "Generic (PLEG): container finished" podID="0180a3e1-bb04-48bd-babd-10eb6eb78853" containerID="537f90b4c7e426d1db41ebb1375a769563a21ddbdb830249a7b8bd01830cf2f9" exitCode=0 Oct 31 01:20:19.184841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.184736 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" event={"ID":"0180a3e1-bb04-48bd-babd-10eb6eb78853","Type":"ContainerDied","Data":"537f90b4c7e426d1db41ebb1375a769563a21ddbdb830249a7b8bd01830cf2f9"} Oct 31 01:20:19.204646 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.204620 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:20:19.209490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.209446 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:20:19.242583 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.242540 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e3ce88-34cf-4afb-8093-dc5809718072-kube-api-access-75jp2\" (UniqueName: \"kubernetes.io/projected/39e3ce88-34cf-4afb-8093-dc5809718072-kube-api-access-75jp2\") pod \"39e3ce88-34cf-4afb-8093-dc5809718072\" (UID: \"39e3ce88-34cf-4afb-8093-dc5809718072\") " Oct 31 01:20:19.242773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.242698 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0180a3e1-bb04-48bd-babd-10eb6eb78853-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a3e1-bb04-48bd-babd-10eb6eb78853-plugins-config-volume\") pod \"0180a3e1-bb04-48bd-babd-10eb6eb78853\" (UID: \"0180a3e1-bb04-48bd-babd-10eb6eb78853\") " Oct 31 01:20:19.242980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.242896 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e3ce88-34cf-4afb-8093-dc5809718072-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e3ce88-34cf-4afb-8093-dc5809718072-plugins-config-volume\") pod \"39e3ce88-34cf-4afb-8093-dc5809718072\" (UID: \"39e3ce88-34cf-4afb-8093-dc5809718072\") " Oct 31 01:20:19.242980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.242950 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0180a3e1-bb04-48bd-babd-10eb6eb78853-kube-api-access-ht6xq\" (UniqueName: \"kubernetes.io/projected/0180a3e1-bb04-48bd-babd-10eb6eb78853-kube-api-access-ht6xq\") pod \"0180a3e1-bb04-48bd-babd-10eb6eb78853\" (UID: \"0180a3e1-bb04-48bd-babd-10eb6eb78853\") " Oct 31 01:20:19.244931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.243279 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e3ce88-34cf-4afb-8093-dc5809718072-plugins-config-volume" pod "39e3ce88-34cf-4afb-8093-dc5809718072" (UID: "39e3ce88-34cf-4afb-8093-dc5809718072"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:19.244931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.243827 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e3ce88-34cf-4afb-8093-dc5809718072-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:19.245761 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.245729 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0180a3e1-bb04-48bd-babd-10eb6eb78853-plugins-config-volume" pod "0180a3e1-bb04-48bd-babd-10eb6eb78853" (UID: "0180a3e1-bb04-48bd-babd-10eb6eb78853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:19.246178 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.246151 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0180a3e1-bb04-48bd-babd-10eb6eb78853-kube-api-access-ht6xq" pod "0180a3e1-bb04-48bd-babd-10eb6eb78853" (UID: "0180a3e1-bb04-48bd-babd-10eb6eb78853"). InnerVolumeSpecName "kube-api-access-ht6xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:19.248929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.248900 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e3ce88-34cf-4afb-8093-dc5809718072-kube-api-access-75jp2" pod "39e3ce88-34cf-4afb-8093-dc5809718072" (UID: "39e3ce88-34cf-4afb-8093-dc5809718072"). InnerVolumeSpecName "kube-api-access-75jp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:19.345386 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.345340 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75jp2\" (UniqueName: \"kubernetes.io/projected/39e3ce88-34cf-4afb-8093-dc5809718072-kube-api-access-75jp2\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:19.345386 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.345383 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0180a3e1-bb04-48bd-babd-10eb6eb78853-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:19.345636 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.345398 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht6xq\" (UniqueName: \"kubernetes.io/projected/0180a3e1-bb04-48bd-babd-10eb6eb78853-kube-api-access-ht6xq\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:19.741155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.741069 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:20:19.957394 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.955977 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:19.969631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:19.969604 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:20:20.195929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.195896 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" Oct 31 01:20:20.196114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.195917 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f" event={"ID":"0180a3e1-bb04-48bd-babd-10eb6eb78853","Type":"ContainerDied","Data":"acc5e7baaade155f1e5119782a3a10f30d514ec1a3931b806eb70fe76c55f872"} Oct 31 01:20:20.196114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.195990 5978 scope.go:122] "RemoveContainer" containerID="537f90b4c7e426d1db41ebb1375a769563a21ddbdb830249a7b8bd01830cf2f9" Oct 31 01:20:20.196322 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.196273 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds" Oct 31 01:20:20.254592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.254556 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f"] Oct 31 01:20:20.265489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.265438 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-114c95-5ff6f64bcf-qbn4f"] Oct 31 01:20:20.352893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.352858 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds"] Oct 31 01:20:20.380822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.380787 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-114c95-795d94c84c-gsbds"] Oct 31 01:20:20.428929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.428895 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0180a3e1-bb04-48bd-babd-10eb6eb78853" path="/var/lib/kubelet/pods/0180a3e1-bb04-48bd-babd-10eb6eb78853/volumes" Oct 31 01:20:20.429305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.429292 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e3ce88-34cf-4afb-8093-dc5809718072" path="/var/lib/kubelet/pods/39e3ce88-34cf-4afb-8093-dc5809718072/volumes" Oct 31 01:20:20.429761 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:20.429746 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52036be9-339e-4bb6-a5c1-30f41c943ff5" path="/var/lib/kubelet/pods/52036be9-339e-4bb6-a5c1-30f41c943ff5/volumes" Oct 31 01:20:21.419623 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:21.419582 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:20:21.420041 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:21.419653 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:21.594223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:21.594179 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" containerID="cri-o://1fa2cb88f3a4c74eeb2fc83e47dd58964ad1e3b73e3fdda862c965088b31c7b1" gracePeriod=45 Oct 31 01:20:21.594409 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:21.594179 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" containerID="cri-o://9479fe65913109e6a0a6904d8e10969b0d1d70b5af25131ed2d7950df15fbfa9" gracePeriod=45 Oct 31 01:20:21.953115 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:21.953014 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": dial tcp 10.134.0.87:8000: connect: connection refused" Oct 31 01:20:22.213248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.213165 5978 generic.go:356] "Generic (PLEG): container finished" podID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerID="9479fe65913109e6a0a6904d8e10969b0d1d70b5af25131ed2d7950df15fbfa9" exitCode=0 Oct 31 01:20:22.213544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.213249 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" event={"ID":"62b7d76f-7f3d-4f53-bc75-840933fde90a","Type":"ContainerDied","Data":"9479fe65913109e6a0a6904d8e10969b0d1d70b5af25131ed2d7950df15fbfa9"} Oct 31 01:20:22.215324 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.215300 5978 generic.go:356] "Generic (PLEG): container finished" podID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerID="1fa2cb88f3a4c74eeb2fc83e47dd58964ad1e3b73e3fdda862c965088b31c7b1" exitCode=0 Oct 31 01:20:22.215504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.215378 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" event={"ID":"dcf4bab9-775a-4571-b7aa-2ec73409539f","Type":"ContainerDied","Data":"1fa2cb88f3a4c74eeb2fc83e47dd58964ad1e3b73e3fdda862c965088b31c7b1"} Oct 31 01:20:22.306167 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.306132 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:22.534017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.533978 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 01:20:22.548318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.548287 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 01:20:22.610252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610202 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-model-cache\") pod \"62b7d76f-7f3d-4f53-bc75-840933fde90a\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " Oct 31 01:20:22.610476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610265 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-home\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-home\") pod \"62b7d76f-7f3d-4f53-bc75-840933fde90a\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " Oct 31 01:20:22.610476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610309 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b7d76f-7f3d-4f53-bc75-840933fde90a-kube-api-access-p9r55\" (UniqueName: \"kubernetes.io/projected/62b7d76f-7f3d-4f53-bc75-840933fde90a-kube-api-access-p9r55\") pod \"62b7d76f-7f3d-4f53-bc75-840933fde90a\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " Oct 31 01:20:22.610476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610346 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-dshm\") pod \"62b7d76f-7f3d-4f53-bc75-840933fde90a\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " Oct 31 01:20:22.610476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610403 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-model-cache\") pod \"dcf4bab9-775a-4571-b7aa-2ec73409539f\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " Oct 31 01:20:22.610739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610446 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-tmp-dir\") pod \"dcf4bab9-775a-4571-b7aa-2ec73409539f\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " Oct 31 01:20:22.610739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610531 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62b7d76f-7f3d-4f53-bc75-840933fde90a-tls-certs\" (UniqueName: \"kubernetes.io/secret/62b7d76f-7f3d-4f53-bc75-840933fde90a-tls-certs\") pod \"62b7d76f-7f3d-4f53-bc75-840933fde90a\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " Oct 31 01:20:22.610739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610588 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-home\") pod \"dcf4bab9-775a-4571-b7aa-2ec73409539f\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " Oct 31 01:20:22.610739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610633 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-model-cache" pod "62b7d76f-7f3d-4f53-bc75-840933fde90a" (UID: "62b7d76f-7f3d-4f53-bc75-840933fde90a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.610739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610686 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf4bab9-775a-4571-b7aa-2ec73409539f-kube-api-access-khxfm\" (UniqueName: \"kubernetes.io/projected/dcf4bab9-775a-4571-b7aa-2ec73409539f-kube-api-access-khxfm\") pod \"dcf4bab9-775a-4571-b7aa-2ec73409539f\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " Oct 31 01:20:22.611025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610751 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf4bab9-775a-4571-b7aa-2ec73409539f-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4bab9-775a-4571-b7aa-2ec73409539f-tls-certs\") pod \"dcf4bab9-775a-4571-b7aa-2ec73409539f\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " Oct 31 01:20:22.611025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610845 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-tmp-dir\") pod \"62b7d76f-7f3d-4f53-bc75-840933fde90a\" (UID: \"62b7d76f-7f3d-4f53-bc75-840933fde90a\") " Oct 31 01:20:22.611025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.610995 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-dshm\") pod \"dcf4bab9-775a-4571-b7aa-2ec73409539f\" (UID: \"dcf4bab9-775a-4571-b7aa-2ec73409539f\") " Oct 31 01:20:22.611755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.611730 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.632395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.632277 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-model-cache" pod "dcf4bab9-775a-4571-b7aa-2ec73409539f" (UID: "dcf4bab9-775a-4571-b7aa-2ec73409539f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.632395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.632363 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-home" pod "dcf4bab9-775a-4571-b7aa-2ec73409539f" (UID: "dcf4bab9-775a-4571-b7aa-2ec73409539f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.633215 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.633154 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-tmp-dir" pod "62b7d76f-7f3d-4f53-bc75-840933fde90a" (UID: "62b7d76f-7f3d-4f53-bc75-840933fde90a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.636267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.634629 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-tmp-dir" pod "dcf4bab9-775a-4571-b7aa-2ec73409539f" (UID: "dcf4bab9-775a-4571-b7aa-2ec73409539f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.636267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.636213 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-dshm" pod "62b7d76f-7f3d-4f53-bc75-840933fde90a" (UID: "62b7d76f-7f3d-4f53-bc75-840933fde90a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.636267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.636226 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-dshm" pod "dcf4bab9-775a-4571-b7aa-2ec73409539f" (UID: "dcf4bab9-775a-4571-b7aa-2ec73409539f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.636267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.636213 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62b7d76f-7f3d-4f53-bc75-840933fde90a-tls-certs" pod "62b7d76f-7f3d-4f53-bc75-840933fde90a" (UID: "62b7d76f-7f3d-4f53-bc75-840933fde90a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:22.636905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.636879 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-home" pod "62b7d76f-7f3d-4f53-bc75-840933fde90a" (UID: "62b7d76f-7f3d-4f53-bc75-840933fde90a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:22.639920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.639884 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf4bab9-775a-4571-b7aa-2ec73409539f-tls-certs" pod "dcf4bab9-775a-4571-b7aa-2ec73409539f" (UID: "dcf4bab9-775a-4571-b7aa-2ec73409539f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:22.643305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.643278 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b7d76f-7f3d-4f53-bc75-840933fde90a-kube-api-access-p9r55" pod "62b7d76f-7f3d-4f53-bc75-840933fde90a" (UID: "62b7d76f-7f3d-4f53-bc75-840933fde90a"). InnerVolumeSpecName "kube-api-access-p9r55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:22.644842 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.644763 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf4bab9-775a-4571-b7aa-2ec73409539f-kube-api-access-khxfm" pod "dcf4bab9-775a-4571-b7aa-2ec73409539f" (UID: "dcf4bab9-775a-4571-b7aa-2ec73409539f"). InnerVolumeSpecName "kube-api-access-khxfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:22.713645 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713611 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713645 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713641 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713645 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713652 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9r55\" (UniqueName: \"kubernetes.io/projected/62b7d76f-7f3d-4f53-bc75-840933fde90a-kube-api-access-p9r55\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713662 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713671 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713718 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713729 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b7d76f-7f3d-4f53-bc75-840933fde90a-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713736 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4bab9-775a-4571-b7aa-2ec73409539f-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713744 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khxfm\" (UniqueName: \"kubernetes.io/projected/dcf4bab9-775a-4571-b7aa-2ec73409539f-kube-api-access-khxfm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713753 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4bab9-775a-4571-b7aa-2ec73409539f-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:22.713990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:22.713762 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b7d76f-7f3d-4f53-bc75-840933fde90a-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:23.226745 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.226707 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" Oct 31 01:20:23.226992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.226719 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx" event={"ID":"62b7d76f-7f3d-4f53-bc75-840933fde90a","Type":"ContainerDied","Data":"2f34b51c4c65b7d30446f5400d7c9784950dd6ffb5faa003aa14883b89c2387a"} Oct 31 01:20:23.226992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.226849 5978 scope.go:122] "RemoveContainer" containerID="9479fe65913109e6a0a6904d8e10969b0d1d70b5af25131ed2d7950df15fbfa9" Oct 31 01:20:23.229541 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.229482 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" event={"ID":"dcf4bab9-775a-4571-b7aa-2ec73409539f","Type":"ContainerDied","Data":"0a184e65cfea33939510816100cc99bc07ed0d3cad39820b9c0439377937dac3"} Oct 31 01:20:23.229641 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.229569 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" Oct 31 01:20:23.290921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.290886 5978 scope.go:122] "RemoveContainer" containerID="1fa2cb88f3a4c74eeb2fc83e47dd58964ad1e3b73e3fdda862c965088b31c7b1" Oct 31 01:20:23.321221 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.321181 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx"] Oct 31 01:20:23.330773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.330739 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-2-w1-b024b5-kserve-5549849fb6-9b4lx"] Oct 31 01:20:23.355517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.355486 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp"] Oct 31 01:20:23.360150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.360115 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp"] Oct 31 01:20:23.391598 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:23.391542 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b024b5/distinct-w1-b024b5-kserve-65cbc4c4cc-4m7rp" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:24.198870 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.198828 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:24.240155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.240115 5978 generic.go:356] "Generic (PLEG): container finished" podID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerID="dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede" exitCode=0 Oct 31 01:20:24.240372 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.240188 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" event={"ID":"03bfd765-42c6-483e-ae9b-df8aff4cff55","Type":"ContainerDied","Data":"dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede"} Oct 31 01:20:24.240758 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.240740 5978 scope.go:122] "RemoveContainer" containerID="dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede" Oct 31 01:20:24.242616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.242588 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:20:24.242746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.242675 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:20:24.242746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.242711 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:20:24.426956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.426918 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" path="/var/lib/kubelet/pods/62b7d76f-7f3d-4f53-bc75-840933fde90a/volumes" Oct 31 01:20:24.427412 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.427396 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" path="/var/lib/kubelet/pods/dcf4bab9-775a-4571-b7aa-2ec73409539f/volumes" Oct 31 01:20:24.688552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.688508 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" podUID="f22077e0-eac8-45ef-92fd-ac6f597563fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:20:24.688735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:24.688618 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" podUID="c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:20:25.253379 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:25.253336 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" event={"ID":"03bfd765-42c6-483e-ae9b-df8aff4cff55","Type":"ContainerStarted","Data":"6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc"} Oct 31 01:20:25.435293 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:25.435259 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:25.580330 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:25.580294 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:25.759000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:25.758967 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt"] Oct 31 01:20:25.759242 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:25.759217 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerName="istio-proxy" containerID="cri-o://aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c" gracePeriod=30 Oct 31 01:20:25.841440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:25.841316 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:25.990475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:25.990083 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:20:25.990475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:25.990260 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 31 01:20:25.990475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:25.990310 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 31 01:20:26.013478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:26.006922 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:26.333331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:26.333277 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:26.972415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:26.972377 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 31 01:20:26.972647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:26.972435 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 31 01:20:27.702472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:27.700133 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:27.972374 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:27.972286 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 31 01:20:27.972374 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:27.972356 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 31 01:20:28.972667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:28.972628 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 31 01:20:28.973088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:28.972688 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 31 01:20:29.542351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:29.542306 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:29.972968 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:29.972929 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 31 01:20:29.973526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:29.972994 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 31 01:20:30.972164 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:30.972129 5978 patch_prober.go:29] interesting pod/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 31 01:20:30.972300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:30.972196 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 31 01:20:31.084215 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.084177 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:20:31.173150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173118 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-token\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-token\") pod \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " Oct 31 01:20:31.173150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173157 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-envoy\") pod \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " Oct 31 01:20:31.173438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173237 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-kube-api-access-z5btn\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-kube-api-access-z5btn\") pod \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " Oct 31 01:20:31.173438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173350 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-credential-socket\") pod \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " Oct 31 01:20:31.173587 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173480 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-certs\") pod \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " Oct 31 01:20:31.173587 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173560 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-podinfo\") pod \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " Oct 31 01:20:31.173711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173637 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-data\") pod \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " Oct 31 01:20:31.173711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173654 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-credential-socket" pod "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" (UID: "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.173711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173674 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-socket\") pod \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " Oct 31 01:20:31.173928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173811 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istiod-ca-cert\") pod \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\" (UID: \"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab\") " Oct 31 01:20:31.173928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.173801 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-certs" pod "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" (UID: "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.174046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.174020 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-data" pod "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" (UID: "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.174216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.174164 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-socket" pod "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" (UID: "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.174354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.174329 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istiod-ca-cert" pod "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" (UID: "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:31.174638 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.174615 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.174730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.174642 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.174730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.174656 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.174730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.174671 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.174730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.174685 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.176341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.175819 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-kube-api-access-z5btn" pod "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" (UID: "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab"). InnerVolumeSpecName "kube-api-access-z5btn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:31.176341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.176144 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-envoy" pod "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" (UID: "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:31.176704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.176678 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-token" pod "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" (UID: "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:31.176840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.176788 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-podinfo" pod "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" (UID: "ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:20:31.229518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.228710 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:31.253352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.253315 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq"] Oct 31 01:20:31.262689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.262653 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z"] Oct 31 01:20:31.272125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.272068 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd"] Oct 31 01:20:31.276266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.276237 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.276266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.276266 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.276558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.276280 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.276558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.276294 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5btn\" (UniqueName: \"kubernetes.io/projected/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab-kube-api-access-z5btn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:31.307949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.307850 5978 generic.go:356] "Generic (PLEG): container finished" podID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerID="aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c" exitCode=0 Oct 31 01:20:31.307949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.307907 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" Oct 31 01:20:31.308223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.307978 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" event={"ID":"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab","Type":"ContainerDied","Data":"aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c"} Oct 31 01:20:31.308223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.308045 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt" event={"ID":"ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab","Type":"ContainerDied","Data":"be8be46f5b98c6aaeb5c95c7fe008fd6d4fd3d1ccbf2ed57b59bebb2f333ab37"} Oct 31 01:20:31.308223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.308066 5978 scope.go:122] "RemoveContainer" containerID="aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c" Oct 31 01:20:31.320433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.320409 5978 scope.go:122] "RemoveContainer" containerID="aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c" Oct 31 01:20:31.320795 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:20:31.320773 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c\": container with ID starting with aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c not found: ID does not exist" containerID="aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c" Oct 31 01:20:31.320877 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.320804 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c"} err="failed to get container status \"aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c\": rpc error: code = NotFound desc = could not find container \"aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c\": container with ID starting with aaab516be470da32f86ba5880912ce254ac56ae68c9a82504c0006488b50c15c not found: ID does not exist" Oct 31 01:20:31.396307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.396274 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:31.403880 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.403850 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt"] Oct 31 01:20:31.410342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.410303 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3c90d8-openshift-default-b8c68db99-pztrt"] Oct 31 01:20:31.977411 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.977378 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:20:31.977862 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.977840 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:20:31.978599 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:31.978580 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:20:32.426530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:32.426493 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" path="/var/lib/kubelet/pods/ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab/volumes" Oct 31 01:20:33.182239 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:20:33.182198 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:20:33.385248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:33.385205 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:34.689293 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:34.689259 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" podUID="f22077e0-eac8-45ef-92fd-ac6f597563fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:20:34.689705 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:34.689261 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" podUID="c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:20:35.086149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:35.086111 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:36.716083 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:36.716044 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:36.970799 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:36.970709 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm"] Oct 31 01:20:37.003298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:37.003267 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6"] Oct 31 01:20:37.147923 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:37.147887 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:37.286402 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:37.286309 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:37.834110 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:37.834080 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 01:20:37.879753 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:37.879721 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 01:20:37.900821 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:37.900781 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-kube-api-access-q8qsl\" (UniqueName: \"kubernetes.io/projected/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-kube-api-access-q8qsl\") pod \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\" (UID: \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\") " Oct 31 01:20:37.901015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:37.900839 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-plugins-config-volume\") pod \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\" (UID: \"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee\") " Oct 31 01:20:37.901319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:37.901291 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-plugins-config-volume" pod "c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" (UID: "c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:37.903234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:37.903205 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-kube-api-access-q8qsl" pod "c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" (UID: "c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee"). InnerVolumeSpecName "kube-api-access-q8qsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:38.002700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.002664 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22077e0-eac8-45ef-92fd-ac6f597563fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22077e0-eac8-45ef-92fd-ac6f597563fa-plugins-config-volume\") pod \"f22077e0-eac8-45ef-92fd-ac6f597563fa\" (UID: \"f22077e0-eac8-45ef-92fd-ac6f597563fa\") " Oct 31 01:20:38.002886 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.002711 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22077e0-eac8-45ef-92fd-ac6f597563fa-kube-api-access-jd77p\" (UniqueName: \"kubernetes.io/projected/f22077e0-eac8-45ef-92fd-ac6f597563fa-kube-api-access-jd77p\") pod \"f22077e0-eac8-45ef-92fd-ac6f597563fa\" (UID: \"f22077e0-eac8-45ef-92fd-ac6f597563fa\") " Oct 31 01:20:38.003068 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.003044 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22077e0-eac8-45ef-92fd-ac6f597563fa-plugins-config-volume" pod "f22077e0-eac8-45ef-92fd-ac6f597563fa" (UID: "f22077e0-eac8-45ef-92fd-ac6f597563fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:38.003288 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.003275 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22077e0-eac8-45ef-92fd-ac6f597563fa-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:38.003327 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.003292 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8qsl\" (UniqueName: \"kubernetes.io/projected/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-kube-api-access-q8qsl\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:38.003327 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.003301 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:38.004825 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.004804 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22077e0-eac8-45ef-92fd-ac6f597563fa-kube-api-access-jd77p" pod "f22077e0-eac8-45ef-92fd-ac6f597563fa" (UID: "f22077e0-eac8-45ef-92fd-ac6f597563fa"). InnerVolumeSpecName "kube-api-access-jd77p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:38.104850 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.104810 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd77p\" (UniqueName: \"kubernetes.io/projected/f22077e0-eac8-45ef-92fd-ac6f597563fa-kube-api-access-jd77p\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:38.361001 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.360912 5978 generic.go:356] "Generic (PLEG): container finished" podID="f22077e0-eac8-45ef-92fd-ac6f597563fa" containerID="6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6" exitCode=0 Oct 31 01:20:38.361001 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.360965 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" Oct 31 01:20:38.361001 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.360993 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" event={"ID":"f22077e0-eac8-45ef-92fd-ac6f597563fa","Type":"ContainerDied","Data":"6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6"} Oct 31 01:20:38.361295 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.361051 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl" event={"ID":"f22077e0-eac8-45ef-92fd-ac6f597563fa","Type":"ContainerDied","Data":"e3ae887ff2a391d90735f99dc543b8d7f4b2fe1c9f83e4b397f1f80c7b0ee215"} Oct 31 01:20:38.361295 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.361075 5978 scope.go:122] "RemoveContainer" containerID="6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6" Oct 31 01:20:38.362728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.362702 5978 generic.go:356] "Generic (PLEG): container finished" podID="c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" containerID="609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df" exitCode=0 Oct 31 01:20:38.362868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.362739 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" Oct 31 01:20:38.362868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.362739 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" event={"ID":"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee","Type":"ContainerDied","Data":"609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df"} Oct 31 01:20:38.362868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.362809 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj" event={"ID":"c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee","Type":"ContainerDied","Data":"1f0b9057c32128721460588ed8e23dc835d591a6d43753147aa91e43f8bc2229"} Oct 31 01:20:38.374805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.374787 5978 scope.go:122] "RemoveContainer" containerID="6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6" Oct 31 01:20:38.375128 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:20:38.375106 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6\": container with ID starting with 6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6 not found: ID does not exist" containerID="6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6" Oct 31 01:20:38.375177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.375138 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6"} err="failed to get container status \"6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6\": rpc error: code = NotFound desc = could not find container \"6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6\": container with ID starting with 6f97358f6767e4dc715f496bf9cb3dc76c9e5b44c57d4e56693f378a048e69b6 not found: ID does not exist" Oct 31 01:20:38.375177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.375157 5978 scope.go:122] "RemoveContainer" containerID="609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df" Oct 31 01:20:38.385151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.385125 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:38.386549 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.386527 5978 scope.go:122] "RemoveContainer" containerID="609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df" Oct 31 01:20:38.386859 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:20:38.386837 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df\": container with ID starting with 609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df not found: ID does not exist" containerID="609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df" Oct 31 01:20:38.386958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.386872 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df"} err="failed to get container status \"609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df\": rpc error: code = NotFound desc = could not find container \"609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df\": container with ID starting with 609c16f0fd8611daefd609c39f3938ad88425d0398c2ab8ce5c33448f508d8df not found: ID does not exist" Oct 31 01:20:38.429590 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.429554 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj"] Oct 31 01:20:38.431293 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.431269 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b024b5-54459f766c-r5fpj"] Oct 31 01:20:38.461099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.461071 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl"] Oct 31 01:20:38.465127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:38.465096 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b024b5-59c565544c-f7gfl"] Oct 31 01:20:39.930258 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:39.930222 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:40.426999 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:40.426967 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" path="/var/lib/kubelet/pods/c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee/volumes" Oct 31 01:20:40.428156 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:40.428136 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22077e0-eac8-45ef-92fd-ac6f597563fa" path="/var/lib/kubelet/pods/f22077e0-eac8-45ef-92fd-ac6f597563fa/volumes" Oct 31 01:20:41.595025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:41.594983 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:41.971649 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:41.971506 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" podUID="44e88831-aaa6-4c0c-96b1-4ab3b07754cc" containerName="payload-processing" containerID="cri-o://daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e" gracePeriod=25 Oct 31 01:20:42.003922 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:42.003872 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" podUID="004ef2cb-1db6-4a6e-90af-70c614728a54" containerName="payload-pre-processing" containerID="cri-o://86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916" gracePeriod=25 Oct 31 01:20:42.789313 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:42.789272 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" podUID="004ef2cb-1db6-4a6e-90af-70c614728a54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:20:42.789749 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:42.789279 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" podUID="44e88831-aaa6-4c0c-96b1-4ab3b07754cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:20:43.263433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:43.263387 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:43.548043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:43.547942 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:43.717905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:43.717862 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:44.505924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:44.505890 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:44.662778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:44.662741 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:44.822876 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:44.822840 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:45.881600 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:45.881571 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:46.279987 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.279903 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" containerID="cri-o://45efc917250b6eed75f20c6944673a620537f0729191a2455f2eafe6233823ca" gracePeriod=45 Oct 31 01:20:46.289905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.289868 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" containerID="cri-o://59eb9d63ad578af786838827064133eb305245956ad948fd219b1a2d63254483" gracePeriod=45 Oct 31 01:20:46.298119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.298085 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" containerID="cri-o://a927de4b0c29935075257f4e70d6f358e092156dd78ac85bcc1fb8044da326b6" gracePeriod=45 Oct 31 01:20:46.353081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.353044 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": dial tcp 10.134.0.79:8000: connect: connection refused" Oct 31 01:20:46.454919 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.454877 5978 generic.go:356] "Generic (PLEG): container finished" podID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerID="45efc917250b6eed75f20c6944673a620537f0729191a2455f2eafe6233823ca" exitCode=0 Oct 31 01:20:46.455171 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.455142 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" event={"ID":"aee1ccd5-48b3-48db-839b-b85f9d78997b","Type":"ContainerDied","Data":"45efc917250b6eed75f20c6944673a620537f0729191a2455f2eafe6233823ca"} Oct 31 01:20:46.459155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.459118 5978 generic.go:356] "Generic (PLEG): container finished" podID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerID="a927de4b0c29935075257f4e70d6f358e092156dd78ac85bcc1fb8044da326b6" exitCode=0 Oct 31 01:20:46.459284 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.459230 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" event={"ID":"bf73fb67-5047-4046-b0f4-c647394f2f05","Type":"ContainerDied","Data":"a927de4b0c29935075257f4e70d6f358e092156dd78ac85bcc1fb8044da326b6"} Oct 31 01:20:46.463085 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.463059 5978 generic.go:356] "Generic (PLEG): container finished" podID="56787030-da65-4d0a-93ce-41ed20e64b19" containerID="59eb9d63ad578af786838827064133eb305245956ad948fd219b1a2d63254483" exitCode=0 Oct 31 01:20:46.463334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.463140 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" event={"ID":"56787030-da65-4d0a-93ce-41ed20e64b19","Type":"ContainerDied","Data":"59eb9d63ad578af786838827064133eb305245956ad948fd219b1a2d63254483"} Oct 31 01:20:46.607765 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.607392 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 01:20:46.661410 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.661374 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 01:20:46.665625 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.665603 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 01:20:46.750752 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.750711 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-dshm\") pod \"aee1ccd5-48b3-48db-839b-b85f9d78997b\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " Oct 31 01:20:46.750930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.750790 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-home\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-home\") pod \"aee1ccd5-48b3-48db-839b-b85f9d78997b\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " Oct 31 01:20:46.750930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.750913 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-model-cache\") pod \"aee1ccd5-48b3-48db-839b-b85f9d78997b\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " Oct 31 01:20:46.751102 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.751082 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee1ccd5-48b3-48db-839b-b85f9d78997b-tls-certs\" (UniqueName: \"kubernetes.io/secret/aee1ccd5-48b3-48db-839b-b85f9d78997b-tls-certs\") pod \"aee1ccd5-48b3-48db-839b-b85f9d78997b\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " Oct 31 01:20:46.751180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.751121 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee1ccd5-48b3-48db-839b-b85f9d78997b-kube-api-access-dbt9h\" (UniqueName: \"kubernetes.io/projected/aee1ccd5-48b3-48db-839b-b85f9d78997b-kube-api-access-dbt9h\") pod \"aee1ccd5-48b3-48db-839b-b85f9d78997b\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " Oct 31 01:20:46.751242 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.751195 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-tmp-dir\") pod \"aee1ccd5-48b3-48db-839b-b85f9d78997b\" (UID: \"aee1ccd5-48b3-48db-839b-b85f9d78997b\") " Oct 31 01:20:46.752062 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.752034 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-tmp-dir" pod "aee1ccd5-48b3-48db-839b-b85f9d78997b" (UID: "aee1ccd5-48b3-48db-839b-b85f9d78997b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.752665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.752640 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-model-cache" pod "aee1ccd5-48b3-48db-839b-b85f9d78997b" (UID: "aee1ccd5-48b3-48db-839b-b85f9d78997b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.752860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.752836 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-home" pod "aee1ccd5-48b3-48db-839b-b85f9d78997b" (UID: "aee1ccd5-48b3-48db-839b-b85f9d78997b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.754863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.754838 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-dshm" pod "aee1ccd5-48b3-48db-839b-b85f9d78997b" (UID: "aee1ccd5-48b3-48db-839b-b85f9d78997b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.755379 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.755351 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee1ccd5-48b3-48db-839b-b85f9d78997b-tls-certs" pod "aee1ccd5-48b3-48db-839b-b85f9d78997b" (UID: "aee1ccd5-48b3-48db-839b-b85f9d78997b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:46.755379 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.755350 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee1ccd5-48b3-48db-839b-b85f9d78997b-kube-api-access-dbt9h" pod "aee1ccd5-48b3-48db-839b-b85f9d78997b" (UID: "aee1ccd5-48b3-48db-839b-b85f9d78997b"). InnerVolumeSpecName "kube-api-access-dbt9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:46.852704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.852608 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56787030-da65-4d0a-93ce-41ed20e64b19-kube-api-access-kmvf5\" (UniqueName: \"kubernetes.io/projected/56787030-da65-4d0a-93ce-41ed20e64b19-kube-api-access-kmvf5\") pod \"56787030-da65-4d0a-93ce-41ed20e64b19\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " Oct 31 01:20:46.852935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.852810 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-dshm\") pod \"56787030-da65-4d0a-93ce-41ed20e64b19\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " Oct 31 01:20:46.852935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.852871 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf73fb67-5047-4046-b0f4-c647394f2f05-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73fb67-5047-4046-b0f4-c647394f2f05-tls-certs\") pod \"bf73fb67-5047-4046-b0f4-c647394f2f05\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " Oct 31 01:20:46.852935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.852898 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-tmp-dir\") pod \"bf73fb67-5047-4046-b0f4-c647394f2f05\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " Oct 31 01:20:46.852935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.852933 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-model-cache\") pod \"bf73fb67-5047-4046-b0f4-c647394f2f05\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " Oct 31 01:20:46.853135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.852965 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf73fb67-5047-4046-b0f4-c647394f2f05-kube-api-access-twggq\" (UniqueName: \"kubernetes.io/projected/bf73fb67-5047-4046-b0f4-c647394f2f05-kube-api-access-twggq\") pod \"bf73fb67-5047-4046-b0f4-c647394f2f05\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " Oct 31 01:20:46.853135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.853055 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56787030-da65-4d0a-93ce-41ed20e64b19-tls-certs\" (UniqueName: \"kubernetes.io/secret/56787030-da65-4d0a-93ce-41ed20e64b19-tls-certs\") pod \"56787030-da65-4d0a-93ce-41ed20e64b19\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " Oct 31 01:20:46.853135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.853096 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-model-cache\") pod \"56787030-da65-4d0a-93ce-41ed20e64b19\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " Oct 31 01:20:46.853135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.853131 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-home\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-home\") pod \"bf73fb67-5047-4046-b0f4-c647394f2f05\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " Oct 31 01:20:46.853268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.853222 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-home\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-home\") pod \"56787030-da65-4d0a-93ce-41ed20e64b19\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " Oct 31 01:20:46.853268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.853248 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-dshm\") pod \"bf73fb67-5047-4046-b0f4-c647394f2f05\" (UID: \"bf73fb67-5047-4046-b0f4-c647394f2f05\") " Oct 31 01:20:46.853349 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.853267 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-tmp-dir\") pod \"56787030-da65-4d0a-93ce-41ed20e64b19\" (UID: \"56787030-da65-4d0a-93ce-41ed20e64b19\") " Oct 31 01:20:46.854023 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.853997 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.854023 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854024 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.854224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854039 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.854224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854053 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee1ccd5-48b3-48db-839b-b85f9d78997b-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.854224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854067 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee1ccd5-48b3-48db-839b-b85f9d78997b-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.854224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854082 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbt9h\" (UniqueName: \"kubernetes.io/projected/aee1ccd5-48b3-48db-839b-b85f9d78997b-kube-api-access-dbt9h\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.855307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854279 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-home" pod "bf73fb67-5047-4046-b0f4-c647394f2f05" (UID: "bf73fb67-5047-4046-b0f4-c647394f2f05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.855307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854329 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-tmp-dir" pod "bf73fb67-5047-4046-b0f4-c647394f2f05" (UID: "bf73fb67-5047-4046-b0f4-c647394f2f05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.855307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854360 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-model-cache" pod "bf73fb67-5047-4046-b0f4-c647394f2f05" (UID: "bf73fb67-5047-4046-b0f4-c647394f2f05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.855307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854755 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-tmp-dir" pod "56787030-da65-4d0a-93ce-41ed20e64b19" (UID: "56787030-da65-4d0a-93ce-41ed20e64b19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.855307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854845 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-model-cache" pod "56787030-da65-4d0a-93ce-41ed20e64b19" (UID: "56787030-da65-4d0a-93ce-41ed20e64b19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.855307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.854999 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-home" pod "56787030-da65-4d0a-93ce-41ed20e64b19" (UID: "56787030-da65-4d0a-93ce-41ed20e64b19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.855307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.855179 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56787030-da65-4d0a-93ce-41ed20e64b19-kube-api-access-kmvf5" pod "56787030-da65-4d0a-93ce-41ed20e64b19" (UID: "56787030-da65-4d0a-93ce-41ed20e64b19"). InnerVolumeSpecName "kube-api-access-kmvf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:46.855706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.855326 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf73fb67-5047-4046-b0f4-c647394f2f05-tls-certs" pod "bf73fb67-5047-4046-b0f4-c647394f2f05" (UID: "bf73fb67-5047-4046-b0f4-c647394f2f05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:46.855706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.855608 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf73fb67-5047-4046-b0f4-c647394f2f05-kube-api-access-twggq" pod "bf73fb67-5047-4046-b0f4-c647394f2f05" (UID: "bf73fb67-5047-4046-b0f4-c647394f2f05"). InnerVolumeSpecName "kube-api-access-twggq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:46.855840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.855825 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-dshm" pod "56787030-da65-4d0a-93ce-41ed20e64b19" (UID: "56787030-da65-4d0a-93ce-41ed20e64b19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.856343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.856324 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-dshm" pod "bf73fb67-5047-4046-b0f4-c647394f2f05" (UID: "bf73fb67-5047-4046-b0f4-c647394f2f05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:46.856594 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.856579 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56787030-da65-4d0a-93ce-41ed20e64b19-tls-certs" pod "56787030-da65-4d0a-93ce-41ed20e64b19" (UID: "56787030-da65-4d0a-93ce-41ed20e64b19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:46.955040 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955003 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955040 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955033 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955040 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955044 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvf5\" (UniqueName: \"kubernetes.io/projected/56787030-da65-4d0a-93ce-41ed20e64b19-kube-api-access-kmvf5\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955053 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955062 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73fb67-5047-4046-b0f4-c647394f2f05-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955070 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955078 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955086 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twggq\" (UniqueName: \"kubernetes.io/projected/bf73fb67-5047-4046-b0f4-c647394f2f05-kube-api-access-twggq\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955094 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56787030-da65-4d0a-93ce-41ed20e64b19-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955113 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955122 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73fb67-5047-4046-b0f4-c647394f2f05-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:46.955529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:46.955129 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56787030-da65-4d0a-93ce-41ed20e64b19-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:47.270065 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.269978 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd"] Oct 31 01:20:47.270697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270653 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f01e023e-3b05-4ca5-a4e2-19c682dcc570" containerName="maas-api" Oct 31 01:20:47.270763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270699 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01e023e-3b05-4ca5-a4e2-19c682dcc570" containerName="maas-api" Oct 31 01:20:47.270763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270711 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f22077e0-eac8-45ef-92fd-ac6f597563fa" containerName="payload-pre-processing" Oct 31 01:20:47.270763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270717 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22077e0-eac8-45ef-92fd-ac6f597563fa" containerName="payload-pre-processing" Oct 31 01:20:47.270763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270731 5978 cpu_manager.go:414] "containerMap: removing container" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" Oct 31 01:20:47.270763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270736 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" Oct 31 01:20:47.270763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270748 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerName="istio-proxy" Oct 31 01:20:47.270763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270753 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerName="istio-proxy" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270768 5978 cpu_manager.go:414] "containerMap: removing container" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270775 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270810 5978 cpu_manager.go:414] "containerMap: removing container" podUID="52036be9-339e-4bb6-a5c1-30f41c943ff5" containerName="maas-api" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270815 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52036be9-339e-4bb6-a5c1-30f41c943ff5" containerName="maas-api" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270830 5978 cpu_manager.go:414] "containerMap: removing container" podUID="39e3ce88-34cf-4afb-8093-dc5809718072" containerName="payload-processing" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270835 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e3ce88-34cf-4afb-8093-dc5809718072" containerName="payload-processing" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270851 5978 cpu_manager.go:414] "containerMap: removing container" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270858 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270869 5978 cpu_manager.go:414] "containerMap: removing container" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270880 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163792a7-efc3-4a2b-96cf-a21b004b7f20" containerName="istio-proxy" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270890 5978 cpu_manager.go:414] "containerMap: removing container" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270895 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270928 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" containerName="payload-processing" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270935 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" containerName="payload-processing" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270947 5978 cpu_manager.go:414] "containerMap: removing container" podUID="0180a3e1-bb04-48bd-babd-10eb6eb78853" containerName="payload-pre-processing" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270954 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0180a3e1-bb04-48bd-babd-10eb6eb78853" containerName="payload-pre-processing" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270972 5978 cpu_manager.go:414] "containerMap: removing container" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" Oct 31 01:20:47.271051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.270977 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271101 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="52036be9-339e-4bb6-a5c1-30f41c943ff5" containerName="maas-api" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271115 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e3ce88-34cf-4afb-8093-dc5809718072" containerName="payload-processing" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271125 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" containerName="main" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271132 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" containerName="main" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271138 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95d49d2-1d4c-49cf-9409-2b6e3bd7c6ee" containerName="payload-processing" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271145 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="0180a3e1-bb04-48bd-babd-10eb6eb78853" containerName="payload-pre-processing" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271151 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b7d76f-7f3d-4f53-bc75-840933fde90a" containerName="main" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271158 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01e023e-3b05-4ca5-a4e2-19c682dcc570" containerName="maas-api" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271164 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22077e0-eac8-45ef-92fd-ac6f597563fa" containerName="payload-pre-processing" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271174 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" containerName="main" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271183 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9cb0b8-c32f-4e4c-bd46-91239c5b45ab" containerName="istio-proxy" Oct 31 01:20:47.271674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.271194 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf4bab9-775a-4571-b7aa-2ec73409539f" containerName="main" Oct 31 01:20:47.274803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.274780 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.287167 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.287144 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:47.287325 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.287307 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:20:47.301510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.301483 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd"] Oct 31 01:20:47.360932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.360896 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc9351a-d9tzd\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.361122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.361003 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdgn8\" (UniqueName: \"kubernetes.io/projected/8352c968-6c31-496d-8ae9-b8c66e31fce0-kube-api-access-pdgn8\") pod \"maas-api-revoke-sub-6dc9351a-d9tzd\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.361122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.361058 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc9351a-d9tzd\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.463348 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.463308 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc9351a-d9tzd\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.463595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.463499 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdgn8\" (UniqueName: \"kubernetes.io/projected/8352c968-6c31-496d-8ae9-b8c66e31fce0-kube-api-access-pdgn8\") pod \"maas-api-revoke-sub-6dc9351a-d9tzd\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.463595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.463558 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc9351a-d9tzd\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.464191 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.464160 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc9351a-d9tzd\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.466681 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.466659 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc9351a-d9tzd\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.473825 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.473806 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" Oct 31 01:20:47.473935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.473839 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd" event={"ID":"bf73fb67-5047-4046-b0f4-c647394f2f05","Type":"ContainerDied","Data":"6211ea78eec3f58808ae9391d3e6380ed807a2d95a834badd62e96572a90ea71"} Oct 31 01:20:47.473935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.473886 5978 scope.go:122] "RemoveContainer" containerID="a927de4b0c29935075257f4e70d6f358e092156dd78ac85bcc1fb8044da326b6" Oct 31 01:20:47.475677 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.475654 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" Oct 31 01:20:47.475827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.475651 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z" event={"ID":"56787030-da65-4d0a-93ce-41ed20e64b19","Type":"ContainerDied","Data":"fa62924d2d6cb3b3af3b4417fdf41094f09f76e6c8c1f4f66ac8556ace92cd04"} Oct 31 01:20:47.477526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.477483 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" event={"ID":"aee1ccd5-48b3-48db-839b-b85f9d78997b","Type":"ContainerDied","Data":"e67ca95a8e5cff31f404b87f6177f69f7c244fe56d7e5d2b6941e0e4ed00d7a9"} Oct 31 01:20:47.477636 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.477552 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq" Oct 31 01:20:47.490030 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.490005 5978 scope.go:122] "RemoveContainer" containerID="59eb9d63ad578af786838827064133eb305245956ad948fd219b1a2d63254483" Oct 31 01:20:47.490796 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.490774 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdgn8\" (UniqueName: \"kubernetes.io/projected/8352c968-6c31-496d-8ae9-b8c66e31fce0-kube-api-access-pdgn8\") pod \"maas-api-revoke-sub-6dc9351a-d9tzd\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.510704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.510681 5978 scope.go:122] "RemoveContainer" containerID="45efc917250b6eed75f20c6944673a620537f0729191a2455f2eafe6233823ca" Oct 31 01:20:47.545782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.545754 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq"] Oct 31 01:20:47.549878 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.549855 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-671959/distinct-w0-671959-kserve-85df5595f4-hbddq"] Oct 31 01:20:47.578498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.578463 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd"] Oct 31 01:20:47.583025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.582997 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-671959/simulator-w0-671959-kserve-74b4fcb8b-2gdvd"] Oct 31 01:20:47.588461 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.588429 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:47.616428 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.616393 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z"] Oct 31 01:20:47.622918 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.622879 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-671959/premium-w0-671959-kserve-5fc47f4fcf-2c48z"] Oct 31 01:20:47.723483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.721839 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:47.893306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:47.893281 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd"] Oct 31 01:20:47.895417 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:20:47.895391 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8352c968_6c31_496d_8ae9_b8c66e31fce0.slice/crio-51bb259d0e90f2443cdfbc47d8335a3213983d9d9bca6de99383a0685dd33eae WatchSource:0}: Error finding container 51bb259d0e90f2443cdfbc47d8335a3213983d9d9bca6de99383a0685dd33eae: Status 404 returned error can't find the container with id 51bb259d0e90f2443cdfbc47d8335a3213983d9d9bca6de99383a0685dd33eae Oct 31 01:20:48.426800 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:48.426770 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56787030-da65-4d0a-93ce-41ed20e64b19" path="/var/lib/kubelet/pods/56787030-da65-4d0a-93ce-41ed20e64b19/volumes" Oct 31 01:20:48.427236 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:48.427221 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee1ccd5-48b3-48db-839b-b85f9d78997b" path="/var/lib/kubelet/pods/aee1ccd5-48b3-48db-839b-b85f9d78997b/volumes" Oct 31 01:20:48.427996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:48.427634 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf73fb67-5047-4046-b0f4-c647394f2f05" path="/var/lib/kubelet/pods/bf73fb67-5047-4046-b0f4-c647394f2f05/volumes" Oct 31 01:20:48.487726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:48.487639 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" event={"ID":"8352c968-6c31-496d-8ae9-b8c66e31fce0","Type":"ContainerStarted","Data":"e86aeb54811cadc950e1a26697141c043dbe83ba96c651f08aa449a4d071b06c"} Oct 31 01:20:48.487726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:48.487702 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" event={"ID":"8352c968-6c31-496d-8ae9-b8c66e31fce0","Type":"ContainerStarted","Data":"51bb259d0e90f2443cdfbc47d8335a3213983d9d9bca6de99383a0685dd33eae"} Oct 31 01:20:48.524242 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:48.524176 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" podStartSLOduration=1.524157744 podStartE2EDuration="1.524157744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:48.52363383 +0000 UTC m=+3666.221747596" watchObservedRunningTime="2026-10-31 01:20:48.524157744 +0000 UTC m=+3666.222271509" Oct 31 01:20:49.280920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:49.273536 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:49.492351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:49.492312 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c"] Oct 31 01:20:49.492860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:49.492529 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" containerID="cri-o://ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2" gracePeriod=30 Oct 31 01:20:49.499000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:49.498974 5978 generic.go:356] "Generic (PLEG): container finished" podID="8352c968-6c31-496d-8ae9-b8c66e31fce0" containerID="e86aeb54811cadc950e1a26697141c043dbe83ba96c651f08aa449a4d071b06c" exitCode=0 Oct 31 01:20:49.499137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:49.499055 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" event={"ID":"8352c968-6c31-496d-8ae9-b8c66e31fce0","Type":"ContainerDied","Data":"e86aeb54811cadc950e1a26697141c043dbe83ba96c651f08aa449a4d071b06c"} Oct 31 01:20:50.314102 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.314067 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:50.655915 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.655867 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:50.691653 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.691626 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:50.708606 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.708576 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-xks-ca\") pod \"8352c968-6c31-496d-8ae9-b8c66e31fce0\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " Oct 31 01:20:50.708840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.708618 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8352c968-6c31-496d-8ae9-b8c66e31fce0-kube-api-access-pdgn8\" (UniqueName: \"kubernetes.io/projected/8352c968-6c31-496d-8ae9-b8c66e31fce0-kube-api-access-pdgn8\") pod \"8352c968-6c31-496d-8ae9-b8c66e31fce0\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " Oct 31 01:20:50.709018 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.708824 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-openshift-ca\") pod \"8352c968-6c31-496d-8ae9-b8c66e31fce0\" (UID: \"8352c968-6c31-496d-8ae9-b8c66e31fce0\") " Oct 31 01:20:50.709344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.709309 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-openshift-ca" pod "8352c968-6c31-496d-8ae9-b8c66e31fce0" (UID: "8352c968-6c31-496d-8ae9-b8c66e31fce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:50.709743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.709714 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:50.711678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.711653 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-xks-ca" pod "8352c968-6c31-496d-8ae9-b8c66e31fce0" (UID: "8352c968-6c31-496d-8ae9-b8c66e31fce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:50.711774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.711682 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8352c968-6c31-496d-8ae9-b8c66e31fce0-kube-api-access-pdgn8" pod "8352c968-6c31-496d-8ae9-b8c66e31fce0" (UID: "8352c968-6c31-496d-8ae9-b8c66e31fce0"). InnerVolumeSpecName "kube-api-access-pdgn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:50.718553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.718530 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc"] Oct 31 01:20:50.718783 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.718761 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" podUID="fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a" containerName="payload-processing" containerID="cri-o://a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782" gracePeriod=30 Oct 31 01:20:50.811132 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.811099 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c968-6c31-496d-8ae9-b8c66e31fce0-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:50.811132 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.811132 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdgn8\" (UniqueName: \"kubernetes.io/projected/8352c968-6c31-496d-8ae9-b8c66e31fce0-kube-api-access-pdgn8\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:50.815145 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.815112 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:50.966316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.966280 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:50.975388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:50.975357 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:20:51.013084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.013055 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-plugins-config-volume\") pod \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\" (UID: \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\") " Oct 31 01:20:51.013261 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.013108 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-kube-api-access-hjzqx\" (UniqueName: \"kubernetes.io/projected/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-kube-api-access-hjzqx\") pod \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\" (UID: \"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a\") " Oct 31 01:20:51.014287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.013663 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-plugins-config-volume" pod "fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a" (UID: "fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:51.014424 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.014322 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:51.015835 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.015800 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-kube-api-access-hjzqx" pod "fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a" (UID: "fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a"). InnerVolumeSpecName "kube-api-access-hjzqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:51.131733 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.131691 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjzqx\" (UniqueName: \"kubernetes.io/projected/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a-kube-api-access-hjzqx\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:51.184652 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.184563 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:51.350547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.350516 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:51.514150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.514062 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" Oct 31 01:20:51.514150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.514086 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd" event={"ID":"8352c968-6c31-496d-8ae9-b8c66e31fce0","Type":"ContainerDied","Data":"51bb259d0e90f2443cdfbc47d8335a3213983d9d9bca6de99383a0685dd33eae"} Oct 31 01:20:51.514150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.514138 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51bb259d0e90f2443cdfbc47d8335a3213983d9d9bca6de99383a0685dd33eae" Oct 31 01:20:51.515809 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.515780 5978 generic.go:356] "Generic (PLEG): container finished" podID="fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a" containerID="a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782" exitCode=0 Oct 31 01:20:51.515954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.515847 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" Oct 31 01:20:51.515954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.515877 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" event={"ID":"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a","Type":"ContainerDied","Data":"a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782"} Oct 31 01:20:51.515954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.515924 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc" event={"ID":"fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a","Type":"ContainerDied","Data":"ff72b0e977e1eb4db0ddc57f8bd0eb852e819d24a55e1a74688ecba76f8e4d52"} Oct 31 01:20:51.515954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.515952 5978 scope.go:122] "RemoveContainer" containerID="a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782" Oct 31 01:20:51.529680 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.529658 5978 scope.go:122] "RemoveContainer" containerID="a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782" Oct 31 01:20:51.529985 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:20:51.529964 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782\": container with ID starting with a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782 not found: ID does not exist" containerID="a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782" Oct 31 01:20:51.530045 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.529994 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782"} err="failed to get container status \"a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782\": rpc error: code = NotFound desc = could not find container \"a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782\": container with ID starting with a7233ff2e54f112a437ff148f27e01c583ebc7990878e5090eb4ba8ac5830782 not found: ID does not exist" Oct 31 01:20:51.575019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.574985 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc"] Oct 31 01:20:51.577776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.577753 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d251f-7fb56dfb5b-9l7xc"] Oct 31 01:20:51.835585 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:51.835558 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:52.427764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:52.427728 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a" path="/var/lib/kubelet/pods/fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a/volumes" Oct 31 01:20:52.789105 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:52.789011 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" podUID="004ef2cb-1db6-4a6e-90af-70c614728a54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:20:52.789269 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:52.789014 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" podUID="44e88831-aaa6-4c0c-96b1-4ab3b07754cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:20:53.090863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:53.090826 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:54.632017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.631977 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:54.791230 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.791201 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:20:54.880167 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880134 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-certs\") pod \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " Oct 31 01:20:54.880392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880183 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-token\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-token\") pod \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " Oct 31 01:20:54.880392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880211 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-credential-socket\") pod \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " Oct 31 01:20:54.880392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880244 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istiod-ca-cert\") pod \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " Oct 31 01:20:54.880392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880272 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-podinfo\") pod \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " Oct 31 01:20:54.880392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880324 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-data\") pod \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " Oct 31 01:20:54.880806 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880541 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-certs" pod "430b24ac-3a49-4be1-badd-5bcebb7c5ef9" (UID: "430b24ac-3a49-4be1-badd-5bcebb7c5ef9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:54.880806 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880561 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-envoy\") pod \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " Oct 31 01:20:54.880806 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880612 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-socket\") pod \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " Oct 31 01:20:54.880806 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880670 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-kube-api-access-mpfrs\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-kube-api-access-mpfrs\") pod \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\" (UID: \"430b24ac-3a49-4be1-badd-5bcebb7c5ef9\") " Oct 31 01:20:54.880806 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.880745 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-credential-socket" pod "430b24ac-3a49-4be1-badd-5bcebb7c5ef9" (UID: "430b24ac-3a49-4be1-badd-5bcebb7c5ef9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:54.881239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.881208 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-socket" pod "430b24ac-3a49-4be1-badd-5bcebb7c5ef9" (UID: "430b24ac-3a49-4be1-badd-5bcebb7c5ef9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:54.881335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.881243 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istiod-ca-cert" pod "430b24ac-3a49-4be1-badd-5bcebb7c5ef9" (UID: "430b24ac-3a49-4be1-badd-5bcebb7c5ef9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:54.881408 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.881344 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-data" pod "430b24ac-3a49-4be1-badd-5bcebb7c5ef9" (UID: "430b24ac-3a49-4be1-badd-5bcebb7c5ef9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:54.882092 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.882068 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:54.882227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.882097 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:54.882227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.882112 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:54.882227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.882125 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:54.882227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.882138 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:54.883233 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.883203 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-token" pod "430b24ac-3a49-4be1-badd-5bcebb7c5ef9" (UID: "430b24ac-3a49-4be1-badd-5bcebb7c5ef9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:54.883393 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.883367 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-podinfo" pod "430b24ac-3a49-4be1-badd-5bcebb7c5ef9" (UID: "430b24ac-3a49-4be1-badd-5bcebb7c5ef9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:20:54.883797 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.883768 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-kube-api-access-mpfrs" pod "430b24ac-3a49-4be1-badd-5bcebb7c5ef9" (UID: "430b24ac-3a49-4be1-badd-5bcebb7c5ef9"). InnerVolumeSpecName "kube-api-access-mpfrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:54.884142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.884119 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-envoy" pod "430b24ac-3a49-4be1-badd-5bcebb7c5ef9" (UID: "430b24ac-3a49-4be1-badd-5bcebb7c5ef9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:54.983647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.983617 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:54.983647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.983643 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:54.983647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.983653 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:54.983969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:54.983661 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfrs\" (UniqueName: \"kubernetes.io/projected/430b24ac-3a49-4be1-badd-5bcebb7c5ef9-kube-api-access-mpfrs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:20:55.545687 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:55.545650 5978 generic.go:356] "Generic (PLEG): container finished" podID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerID="ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2" exitCode=0 Oct 31 01:20:55.545897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:55.545700 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" Oct 31 01:20:55.545897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:55.545746 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" event={"ID":"430b24ac-3a49-4be1-badd-5bcebb7c5ef9","Type":"ContainerDied","Data":"ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2"} Oct 31 01:20:55.545897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:55.545799 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c" event={"ID":"430b24ac-3a49-4be1-badd-5bcebb7c5ef9","Type":"ContainerDied","Data":"d610086524f42bc14c1e37c8b99b26fac5402b864b6c2d4de3fecac055344cf8"} Oct 31 01:20:55.545897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:55.545820 5978 scope.go:122] "RemoveContainer" containerID="ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2" Oct 31 01:20:55.557550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:55.557528 5978 scope.go:122] "RemoveContainer" containerID="ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2" Oct 31 01:20:55.557996 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:20:55.557961 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2\": container with ID starting with ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2 not found: ID does not exist" containerID="ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2" Oct 31 01:20:55.558119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:55.558004 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2"} err="failed to get container status \"ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2\": rpc error: code = NotFound desc = could not find container \"ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2\": container with ID starting with ae793b04fb0433ef6a636859aa2f7b6ff48583937d6033ec441094bbe3d27bd2 not found: ID does not exist" Oct 31 01:20:55.605709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:55.605676 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c"] Oct 31 01:20:55.618488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:55.618443 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-714f8f-openshift-default-85496857c7-cdl2c"] Oct 31 01:20:56.140777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:56.140678 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:56.426413 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:56.426337 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" path="/var/lib/kubelet/pods/430b24ac-3a49-4be1-badd-5bcebb7c5ef9/volumes" Oct 31 01:20:56.969480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:56.969413 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds"] Oct 31 01:20:56.969682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:56.969658 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" containerID="cri-o://3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31" gracePeriod=30 Oct 31 01:20:57.782955 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:57.782915 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:58.860022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:58.859987 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:59.006104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:59.006059 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:59.132219 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:59.132129 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr"] Oct 31 01:20:59.132389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:59.132355 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" containerID="cri-o://b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9" gracePeriod=30 Oct 31 01:20:59.200946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:59.200900 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:59.514490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:20:59.511255 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:01.648867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:01.648826 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:01.787592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:01.787560 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:02.094195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.094163 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb"] Oct 31 01:21:02.094916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.094894 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8352c968-6c31-496d-8ae9-b8c66e31fce0" containerName="revoke-keys" Oct 31 01:21:02.094916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.094913 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8352c968-6c31-496d-8ae9-b8c66e31fce0" containerName="revoke-keys" Oct 31 01:21:02.095125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.094926 5978 cpu_manager.go:414] "containerMap: removing container" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" Oct 31 01:21:02.095125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.094934 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" Oct 31 01:21:02.095125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.094948 5978 cpu_manager.go:414] "containerMap: removing container" podUID="fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a" containerName="payload-processing" Oct 31 01:21:02.095125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.094953 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a" containerName="payload-processing" Oct 31 01:21:02.095125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.095079 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8352c968-6c31-496d-8ae9-b8c66e31fce0" containerName="revoke-keys" Oct 31 01:21:02.095125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.095093 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="430b24ac-3a49-4be1-badd-5bcebb7c5ef9" containerName="istio-proxy" Oct 31 01:21:02.095125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.095102 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0a8c9f-dfe2-4a7d-b4f7-bbc869ae5f6a" containerName="payload-processing" Oct 31 01:21:02.099514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.099489 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.107023 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.106994 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:21:02.107149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.107064 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:21:02.115927 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.115899 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8f0029-openshift-default-dockercfg-g7gfd\"" Oct 31 01:21:02.142423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.142393 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb"] Oct 31 01:21:02.197511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.196310 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.197511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.196502 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.197511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.196552 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.197511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.196671 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.197511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.196704 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.197511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.196751 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.197511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.196796 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.197511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.196885 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3feb663-8df2-41f4-acfa-4f883edb3740-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.197511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.196916 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qsb\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-kube-api-access-q4qsb\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.315718 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.315822 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3feb663-8df2-41f4-acfa-4f883edb3740-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.315854 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qsb\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-kube-api-access-q4qsb\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.315923 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.315992 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.316035 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.316152 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.316182 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.316227 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.316690 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.316706 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.317311 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.317590 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.318139 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3feb663-8df2-41f4-acfa-4f883edb3740-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.318347 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:02.326254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.321309 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.330728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.330696 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.353774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.352493 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qsb\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-kube-api-access-q4qsb\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.396774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.361923 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.423744 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.423714 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 01:21:02.437850 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.437824 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:21:02.441258 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.441238 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 01:21:02.472682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.472645 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:02.478798 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.478763 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:02.518248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518217 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-token\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-token\") pod \"147b0ded-3105-4811-ad7d-ab09acd4cb30\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " Oct 31 01:21:02.518419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518320 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147b0ded-3105-4811-ad7d-ab09acd4cb30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b0ded-3105-4811-ad7d-ab09acd4cb30-istiod-ca-cert\") pod \"147b0ded-3105-4811-ad7d-ab09acd4cb30\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " Oct 31 01:21:02.518419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518361 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004ef2cb-1db6-4a6e-90af-70c614728a54-kube-api-access-hq7j2\" (UniqueName: \"kubernetes.io/projected/004ef2cb-1db6-4a6e-90af-70c614728a54-kube-api-access-hq7j2\") pod \"004ef2cb-1db6-4a6e-90af-70c614728a54\" (UID: \"004ef2cb-1db6-4a6e-90af-70c614728a54\") " Oct 31 01:21:02.518419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518387 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004ef2cb-1db6-4a6e-90af-70c614728a54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004ef2cb-1db6-4a6e-90af-70c614728a54-plugins-config-volume\") pod \"004ef2cb-1db6-4a6e-90af-70c614728a54\" (UID: \"004ef2cb-1db6-4a6e-90af-70c614728a54\") " Oct 31 01:21:02.518656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518497 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-plugins-config-volume\") pod \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\" (UID: \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\") " Oct 31 01:21:02.518656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518557 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-podinfo\") pod \"147b0ded-3105-4811-ad7d-ab09acd4cb30\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " Oct 31 01:21:02.518656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518597 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-kube-api-access-664ss\" (UniqueName: \"kubernetes.io/projected/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-kube-api-access-664ss\") pod \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\" (UID: \"44e88831-aaa6-4c0c-96b1-4ab3b07754cc\") " Oct 31 01:21:02.518940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518897 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-plugins-config-volume" pod "44e88831-aaa6-4c0c-96b1-4ab3b07754cc" (UID: "44e88831-aaa6-4c0c-96b1-4ab3b07754cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:02.518940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518896 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-certs\") pod \"147b0ded-3105-4811-ad7d-ab09acd4cb30\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " Oct 31 01:21:02.519209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.518969 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147b0ded-3105-4811-ad7d-ab09acd4cb30-istiod-ca-cert" pod "147b0ded-3105-4811-ad7d-ab09acd4cb30" (UID: "147b0ded-3105-4811-ad7d-ab09acd4cb30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:02.519209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.519029 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-credential-socket\") pod \"147b0ded-3105-4811-ad7d-ab09acd4cb30\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " Oct 31 01:21:02.519209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.519085 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-kube-api-access-v4zll\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-kube-api-access-v4zll\") pod \"147b0ded-3105-4811-ad7d-ab09acd4cb30\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " Oct 31 01:21:02.519209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.519094 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004ef2cb-1db6-4a6e-90af-70c614728a54-plugins-config-volume" pod "004ef2cb-1db6-4a6e-90af-70c614728a54" (UID: "004ef2cb-1db6-4a6e-90af-70c614728a54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:02.519209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.519158 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-data\") pod \"147b0ded-3105-4811-ad7d-ab09acd4cb30\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " Oct 31 01:21:02.519707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.519365 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-envoy\") pod \"147b0ded-3105-4811-ad7d-ab09acd4cb30\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " Oct 31 01:21:02.519707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.519374 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-credential-socket" pod "147b0ded-3105-4811-ad7d-ab09acd4cb30" (UID: "147b0ded-3105-4811-ad7d-ab09acd4cb30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:02.519707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.519423 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-socket\") pod \"147b0ded-3105-4811-ad7d-ab09acd4cb30\" (UID: \"147b0ded-3105-4811-ad7d-ab09acd4cb30\") " Oct 31 01:21:02.519998 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.519800 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-socket" pod "147b0ded-3105-4811-ad7d-ab09acd4cb30" (UID: "147b0ded-3105-4811-ad7d-ab09acd4cb30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:02.519998 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.519927 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-certs" pod "147b0ded-3105-4811-ad7d-ab09acd4cb30" (UID: "147b0ded-3105-4811-ad7d-ab09acd4cb30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.520290 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-data" pod "147b0ded-3105-4811-ad7d-ab09acd4cb30" (UID: "147b0ded-3105-4811-ad7d-ab09acd4cb30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.521055 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.521530 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.521548 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.521558 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147b0ded-3105-4811-ad7d-ab09acd4cb30-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.521568 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004ef2cb-1db6-4a6e-90af-70c614728a54-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.521576 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.521584 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.522052 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-kube-api-access-v4zll" pod "147b0ded-3105-4811-ad7d-ab09acd4cb30" (UID: "147b0ded-3105-4811-ad7d-ab09acd4cb30"). InnerVolumeSpecName "kube-api-access-v4zll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.522646 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-podinfo" pod "147b0ded-3105-4811-ad7d-ab09acd4cb30" (UID: "147b0ded-3105-4811-ad7d-ab09acd4cb30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.524140 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004ef2cb-1db6-4a6e-90af-70c614728a54-kube-api-access-hq7j2" pod "004ef2cb-1db6-4a6e-90af-70c614728a54" (UID: "004ef2cb-1db6-4a6e-90af-70c614728a54"). InnerVolumeSpecName "kube-api-access-hq7j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:02.526682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.524933 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-token" pod "147b0ded-3105-4811-ad7d-ab09acd4cb30" (UID: "147b0ded-3105-4811-ad7d-ab09acd4cb30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:02.528891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.528866 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-kube-api-access-664ss" pod "44e88831-aaa6-4c0c-96b1-4ab3b07754cc" (UID: "44e88831-aaa6-4c0c-96b1-4ab3b07754cc"). InnerVolumeSpecName "kube-api-access-664ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:02.529040 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.528978 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-envoy" pod "147b0ded-3105-4811-ad7d-ab09acd4cb30" (UID: "147b0ded-3105-4811-ad7d-ab09acd4cb30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:02.594947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.594910 5978 generic.go:356] "Generic (PLEG): container finished" podID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerID="3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31" exitCode=0 Oct 31 01:21:02.595076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.594961 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" Oct 31 01:21:02.597163 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.594965 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" event={"ID":"147b0ded-3105-4811-ad7d-ab09acd4cb30","Type":"ContainerDied","Data":"3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31"} Oct 31 01:21:02.597237 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.597214 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds" event={"ID":"147b0ded-3105-4811-ad7d-ab09acd4cb30","Type":"ContainerDied","Data":"b87fd5cd6208188b6593d075c80bf0f7848e8d4139e973537c06f2279b8d5b83"} Oct 31 01:21:02.597279 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.597259 5978 scope.go:122] "RemoveContainer" containerID="3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31" Oct 31 01:21:02.599265 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.599236 5978 generic.go:356] "Generic (PLEG): container finished" podID="004ef2cb-1db6-4a6e-90af-70c614728a54" containerID="86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916" exitCode=0 Oct 31 01:21:02.599372 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.599331 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" Oct 31 01:21:02.599655 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.599544 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" event={"ID":"004ef2cb-1db6-4a6e-90af-70c614728a54","Type":"ContainerDied","Data":"86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916"} Oct 31 01:21:02.599655 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.599598 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6" event={"ID":"004ef2cb-1db6-4a6e-90af-70c614728a54","Type":"ContainerDied","Data":"1543f2f32c84970345abb73c0f8e0807e6bf6f033991f8185650fb71b9baf980"} Oct 31 01:21:02.601685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.601663 5978 generic.go:356] "Generic (PLEG): container finished" podID="44e88831-aaa6-4c0c-96b1-4ab3b07754cc" containerID="daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e" exitCode=0 Oct 31 01:21:02.601808 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.601722 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" event={"ID":"44e88831-aaa6-4c0c-96b1-4ab3b07754cc","Type":"ContainerDied","Data":"daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e"} Oct 31 01:21:02.601808 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.601760 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" event={"ID":"44e88831-aaa6-4c0c-96b1-4ab3b07754cc","Type":"ContainerDied","Data":"beffc5d6f1a99eeedd3ca910d9ab2e5922436e74a4ac617bbb68b74f3d510d8c"} Oct 31 01:21:02.601939 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.601823 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm" Oct 31 01:21:02.621269 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.621241 5978 scope.go:122] "RemoveContainer" containerID="3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31" Oct 31 01:21:02.621940 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:02.621909 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31\": container with ID starting with 3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31 not found: ID does not exist" containerID="3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31" Oct 31 01:21:02.622107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.621950 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31"} err="failed to get container status \"3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31\": rpc error: code = NotFound desc = could not find container \"3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31\": container with ID starting with 3f06d5de5991fd5bc43a927173d6aa263a0eb352928ce26e8b5741759818de31 not found: ID does not exist" Oct 31 01:21:02.622107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.621977 5978 scope.go:122] "RemoveContainer" containerID="86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916" Oct 31 01:21:02.623542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.622727 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.623542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.622774 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.623542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.622789 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq7j2\" (UniqueName: \"kubernetes.io/projected/004ef2cb-1db6-4a6e-90af-70c614728a54-kube-api-access-hq7j2\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.623542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.622803 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147b0ded-3105-4811-ad7d-ab09acd4cb30-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.623542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.622818 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-664ss\" (UniqueName: \"kubernetes.io/projected/44e88831-aaa6-4c0c-96b1-4ab3b07754cc-kube-api-access-664ss\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.623542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.622833 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zll\" (UniqueName: \"kubernetes.io/projected/147b0ded-3105-4811-ad7d-ab09acd4cb30-kube-api-access-v4zll\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:02.637065 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.636601 5978 scope.go:122] "RemoveContainer" containerID="86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916" Oct 31 01:21:02.638706 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:02.638528 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916\": container with ID starting with 86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916 not found: ID does not exist" containerID="86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916" Oct 31 01:21:02.638706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.638568 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916"} err="failed to get container status \"86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916\": rpc error: code = NotFound desc = could not find container \"86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916\": container with ID starting with 86e088def4687436afadcea0386b1971c49fb3c761649e90ed443916bc18d916 not found: ID does not exist" Oct 31 01:21:02.638706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.638601 5978 scope.go:122] "RemoveContainer" containerID="daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e" Oct 31 01:21:02.651820 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.651797 5978 scope.go:122] "RemoveContainer" containerID="daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e" Oct 31 01:21:02.652220 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:02.652192 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e\": container with ID starting with daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e not found: ID does not exist" containerID="daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e" Oct 31 01:21:02.652301 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.652232 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e"} err="failed to get container status \"daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e\": rpc error: code = NotFound desc = could not find container \"daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e\": container with ID starting with daacb535083a7c00d79be896227e06a117e37306e70d63a6f7d7a017978f9c3e not found: ID does not exist" Oct 31 01:21:02.671373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.671343 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm"] Oct 31 01:21:02.676045 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.676020 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-671959-6bb586d9d5-zwlhm"] Oct 31 01:21:02.705589 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.705560 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds"] Oct 31 01:21:02.712701 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.712673 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-328d61-openshift-default-7857fc774c-g49ds"] Oct 31 01:21:02.741208 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.741169 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6"] Oct 31 01:21:02.745961 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.745935 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-671959-86968d9675-tpkw6"] Oct 31 01:21:02.828767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.828719 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb"] Oct 31 01:21:02.831683 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:21:02.831656 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3feb663_8df2_41f4_acfa_4f883edb3740.slice/crio-58a49ccb62300eb887e40cbd9a70de12c41423b6350aa02df8ce36217af45314 WatchSource:0}: Error finding container 58a49ccb62300eb887e40cbd9a70de12c41423b6350aa02df8ce36217af45314: Status 404 returned error can't find the container with id 58a49ccb62300eb887e40cbd9a70de12c41423b6350aa02df8ce36217af45314 Oct 31 01:21:02.834299 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.834269 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:21:02.834379 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.834335 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:21:02.834379 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:02.834371 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:21:03.191844 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:03.191751 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:21:03.610524 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:03.610440 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" event={"ID":"b3feb663-8df2-41f4-acfa-4f883edb3740","Type":"ContainerStarted","Data":"c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591"} Oct 31 01:21:03.610755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:03.610547 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" event={"ID":"b3feb663-8df2-41f4-acfa-4f883edb3740","Type":"ContainerStarted","Data":"58a49ccb62300eb887e40cbd9a70de12c41423b6350aa02df8ce36217af45314"} Oct 31 01:21:03.652912 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:03.652859 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" podStartSLOduration=1.652844318 podStartE2EDuration="1.652844318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:03.651931989 +0000 UTC m=+3681.350045754" watchObservedRunningTime="2026-10-31 01:21:03.652844318 +0000 UTC m=+3681.350958075" Oct 31 01:21:03.709121 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:03.709070 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:04.431754 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.431669 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004ef2cb-1db6-4a6e-90af-70c614728a54" path="/var/lib/kubelet/pods/004ef2cb-1db6-4a6e-90af-70c614728a54/volumes" Oct 31 01:21:04.432174 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.432156 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" path="/var/lib/kubelet/pods/147b0ded-3105-4811-ad7d-ab09acd4cb30/volumes" Oct 31 01:21:04.432595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.432578 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e88831-aaa6-4c0c-96b1-4ab3b07754cc" path="/var/lib/kubelet/pods/44e88831-aaa6-4c0c-96b1-4ab3b07754cc/volumes" Oct 31 01:21:04.478962 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.478932 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:04.482828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.482794 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:21:04.485928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.485904 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:04.748537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.716263 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-data\") pod \"20d2df58-6860-403f-acb9-1b28d74a7791\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " Oct 31 01:21:04.748537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.716347 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-kube-api-access-jfqq7\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-kube-api-access-jfqq7\") pod \"20d2df58-6860-403f-acb9-1b28d74a7791\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " Oct 31 01:21:04.748537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.716406 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-credential-socket\") pod \"20d2df58-6860-403f-acb9-1b28d74a7791\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " Oct 31 01:21:04.748537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.716470 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20d2df58-6860-403f-acb9-1b28d74a7791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d2df58-6860-403f-acb9-1b28d74a7791-istio-podinfo\") pod \"20d2df58-6860-403f-acb9-1b28d74a7791\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " Oct 31 01:21:04.748537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.716512 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-certs\") pod \"20d2df58-6860-403f-acb9-1b28d74a7791\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " Oct 31 01:21:04.748537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.716549 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-socket\") pod \"20d2df58-6860-403f-acb9-1b28d74a7791\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " Oct 31 01:21:04.748537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.716738 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-envoy\") pod \"20d2df58-6860-403f-acb9-1b28d74a7791\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " Oct 31 01:21:04.748537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.716802 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d2df58-6860-403f-acb9-1b28d74a7791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d2df58-6860-403f-acb9-1b28d74a7791-istiod-ca-cert\") pod \"20d2df58-6860-403f-acb9-1b28d74a7791\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " Oct 31 01:21:04.748537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.716840 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-istio-token\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-istio-token\") pod \"20d2df58-6860-403f-acb9-1b28d74a7791\" (UID: \"20d2df58-6860-403f-acb9-1b28d74a7791\") " Oct 31 01:21:04.809384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.800244 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-data" pod "20d2df58-6860-403f-acb9-1b28d74a7791" (UID: "20d2df58-6860-403f-acb9-1b28d74a7791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:04.809384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.800507 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d2df58-6860-403f-acb9-1b28d74a7791-istiod-ca-cert" pod "20d2df58-6860-403f-acb9-1b28d74a7791" (UID: "20d2df58-6860-403f-acb9-1b28d74a7791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:04.809384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.800815 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-socket" pod "20d2df58-6860-403f-acb9-1b28d74a7791" (UID: "20d2df58-6860-403f-acb9-1b28d74a7791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:04.809384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.800889 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-credential-socket" pod "20d2df58-6860-403f-acb9-1b28d74a7791" (UID: "20d2df58-6860-403f-acb9-1b28d74a7791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:04.809384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.808970 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-certs" pod "20d2df58-6860-403f-acb9-1b28d74a7791" (UID: "20d2df58-6860-403f-acb9-1b28d74a7791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:04.825934 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.825894 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d2df58-6860-403f-acb9-1b28d74a7791-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:04.826171 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.826156 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:04.826272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.826259 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:04.826353 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.826343 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:04.826436 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.826426 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:04.827123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.827080 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-envoy" pod "20d2df58-6860-403f-acb9-1b28d74a7791" (UID: "20d2df58-6860-403f-acb9-1b28d74a7791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:04.828398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.827651 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20d2df58-6860-403f-acb9-1b28d74a7791-istio-podinfo" pod "20d2df58-6860-403f-acb9-1b28d74a7791" (UID: "20d2df58-6860-403f-acb9-1b28d74a7791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:21:04.838946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.838902 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-istio-token" pod "20d2df58-6860-403f-acb9-1b28d74a7791" (UID: "20d2df58-6860-403f-acb9-1b28d74a7791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:04.842350 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.842308 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-kube-api-access-jfqq7" pod "20d2df58-6860-403f-acb9-1b28d74a7791" (UID: "20d2df58-6860-403f-acb9-1b28d74a7791"). InnerVolumeSpecName "kube-api-access-jfqq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:04.853657 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.852834 5978 generic.go:356] "Generic (PLEG): container finished" podID="20d2df58-6860-403f-acb9-1b28d74a7791" containerID="b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9" exitCode=0 Oct 31 01:21:04.853657 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.853573 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" Oct 31 01:21:04.857155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.854666 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" event={"ID":"20d2df58-6860-403f-acb9-1b28d74a7791","Type":"ContainerDied","Data":"b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9"} Oct 31 01:21:04.857155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.854727 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:04.857155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.854755 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr" event={"ID":"20d2df58-6860-403f-acb9-1b28d74a7791","Type":"ContainerDied","Data":"e20ecf86590d7ee6c707f717be3bff5384ce6a110b0cd8537f974c0fa96fd96d"} Oct 31 01:21:04.857155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.854786 5978 scope.go:122] "RemoveContainer" containerID="b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9" Oct 31 01:21:04.857155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.856606 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:21:04.884074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.881366 5978 scope.go:122] "RemoveContainer" containerID="d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f" Oct 31 01:21:04.895782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.895757 5978 scope.go:122] "RemoveContainer" containerID="b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9" Oct 31 01:21:04.896292 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:04.896250 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9\": container with ID starting with b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9 not found: ID does not exist" containerID="b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9" Oct 31 01:21:04.896427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.896290 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9"} err="failed to get container status \"b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9\": rpc error: code = NotFound desc = could not find container \"b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9\": container with ID starting with b21013f2e8ff8abcea1b568b7a305d3c720fa2a9bc0be299894e176d648d5fa9 not found: ID does not exist" Oct 31 01:21:04.896427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.896309 5978 scope.go:122] "RemoveContainer" containerID="d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f" Oct 31 01:21:04.896937 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:04.896908 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f\": container with ID starting with d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f not found: ID does not exist" containerID="d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f" Oct 31 01:21:04.897032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.896938 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f"} err="failed to get container status \"d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f\": rpc error: code = NotFound desc = could not find container \"d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f\": container with ID starting with d5d50cb51c6812f58dae067f16ee5658111756330b17295b8751140810df618f not found: ID does not exist" Oct 31 01:21:04.928508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.928291 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d2df58-6860-403f-acb9-1b28d74a7791-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:04.928508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.928342 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:04.928508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.928373 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfqq7\" (UniqueName: \"kubernetes.io/projected/20d2df58-6860-403f-acb9-1b28d74a7791-kube-api-access-jfqq7\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:04.928508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.928390 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d2df58-6860-403f-acb9-1b28d74a7791-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:04.955764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.955729 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr"] Oct 31 01:21:04.964812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:04.964776 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8d251f-openshift-default-7c4f9bdf7-dz6kr"] Oct 31 01:21:05.460603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:05.460510 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:06.169007 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:06.168968 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:06.325107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:06.325073 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:06.426523 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:06.426411 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" path="/var/lib/kubelet/pods/20d2df58-6860-403f-acb9-1b28d74a7791/volumes" Oct 31 01:21:07.181367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.181329 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:07.855712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.855670 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:07.997748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.997711 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2"] Oct 31 01:21:07.998334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998319 5978 cpu_manager.go:414] "containerMap: removing container" podUID="004ef2cb-1db6-4a6e-90af-70c614728a54" containerName="payload-pre-processing" Oct 31 01:21:07.998376 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998336 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004ef2cb-1db6-4a6e-90af-70c614728a54" containerName="payload-pre-processing" Oct 31 01:21:07.998414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998377 5978 cpu_manager.go:414] "containerMap: removing container" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" Oct 31 01:21:07.998414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998384 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" Oct 31 01:21:07.998414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998399 5978 cpu_manager.go:414] "containerMap: removing container" podUID="44e88831-aaa6-4c0c-96b1-4ab3b07754cc" containerName="payload-processing" Oct 31 01:21:07.998414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998404 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e88831-aaa6-4c0c-96b1-4ab3b07754cc" containerName="payload-processing" Oct 31 01:21:07.998414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998412 5978 cpu_manager.go:414] "containerMap: removing container" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" Oct 31 01:21:07.998635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998418 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" Oct 31 01:21:07.998635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998425 5978 cpu_manager.go:414] "containerMap: removing container" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" Oct 31 01:21:07.998635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998430 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" Oct 31 01:21:07.998635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998566 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="147b0ded-3105-4811-ad7d-ab09acd4cb30" containerName="istio-proxy" Oct 31 01:21:07.998635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998590 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e88831-aaa6-4c0c-96b1-4ab3b07754cc" containerName="payload-processing" Oct 31 01:21:07.998635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998598 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" Oct 31 01:21:07.998635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998606 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d2df58-6860-403f-acb9-1b28d74a7791" containerName="istio-proxy" Oct 31 01:21:07.998635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:07.998615 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="004ef2cb-1db6-4a6e-90af-70c614728a54" containerName="payload-pre-processing" Oct 31 01:21:08.003670 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.003649 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:08.005973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.005947 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:08.029526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.029496 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8f0029\"" Oct 31 01:21:08.029702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.029555 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8f0029-dockercfg-2xk7r\"" Oct 31 01:21:08.033742 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.033704 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2"] Oct 31 01:21:08.083516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.083474 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9b719-1a5c-46b5-a49d-e0d3103c758e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2\" (UID: \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:08.083719 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.083537 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwh5\" (UniqueName: \"kubernetes.io/projected/5da9b719-1a5c-46b5-a49d-e0d3103c758e-kube-api-access-mrwh5\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2\" (UID: \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:08.143537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.140033 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5"] Oct 31 01:21:08.164352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.160177 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5"] Oct 31 01:21:08.164704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.160707 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:08.185071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.185039 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9b719-1a5c-46b5-a49d-e0d3103c758e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2\" (UID: \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:08.185421 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.185084 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwh5\" (UniqueName: \"kubernetes.io/projected/5da9b719-1a5c-46b5-a49d-e0d3103c758e-kube-api-access-mrwh5\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2\" (UID: \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:08.185783 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.185758 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9b719-1a5c-46b5-a49d-e0d3103c758e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2\" (UID: \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:08.206570 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.206541 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwh5\" (UniqueName: \"kubernetes.io/projected/5da9b719-1a5c-46b5-a49d-e0d3103c758e-kube-api-access-mrwh5\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2\" (UID: \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:08.286126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.286091 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwp76\" (UniqueName: \"kubernetes.io/projected/612909f8-0549-482d-bff0-8c961a462eae-kube-api-access-jwp76\") pod \"payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5\" (UID: \"612909f8-0549-482d-bff0-8c961a462eae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:08.286338 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.286234 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612909f8-0549-482d-bff0-8c961a462eae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5\" (UID: \"612909f8-0549-482d-bff0-8c961a462eae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:08.317841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.317803 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:08.387485 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.387381 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612909f8-0549-482d-bff0-8c961a462eae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5\" (UID: \"612909f8-0549-482d-bff0-8c961a462eae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:08.390013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.387691 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwp76\" (UniqueName: \"kubernetes.io/projected/612909f8-0549-482d-bff0-8c961a462eae-kube-api-access-jwp76\") pod \"payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5\" (UID: \"612909f8-0549-482d-bff0-8c961a462eae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:08.390013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.388767 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612909f8-0549-482d-bff0-8c961a462eae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5\" (UID: \"612909f8-0549-482d-bff0-8c961a462eae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:08.408960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.408891 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwp76\" (UniqueName: \"kubernetes.io/projected/612909f8-0549-482d-bff0-8c961a462eae-kube-api-access-jwp76\") pod \"payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5\" (UID: \"612909f8-0549-482d-bff0-8c961a462eae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:08.485290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.485254 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:08.620837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.620804 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2"] Oct 31 01:21:08.631573 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:21:08.631270 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da9b719_1a5c_46b5_a49d_e0d3103c758e.slice/crio-170c547ddb7292d8eab38d3ba2009c080efd52080afbbb830fdccc43c0768214 WatchSource:0}: Error finding container 170c547ddb7292d8eab38d3ba2009c080efd52080afbbb830fdccc43c0768214: Status 404 returned error can't find the container with id 170c547ddb7292d8eab38d3ba2009c080efd52080afbbb830fdccc43c0768214 Oct 31 01:21:08.653275 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.648571 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:08.808484 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.808442 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5"] Oct 31 01:21:08.810748 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:21:08.810708 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612909f8_0549_482d_bff0_8c961a462eae.slice/crio-443872558dfe612d5317eefb9b1b1ce13a018b14230d0ae0b41793eb4ecc2238 WatchSource:0}: Error finding container 443872558dfe612d5317eefb9b1b1ce13a018b14230d0ae0b41793eb4ecc2238: Status 404 returned error can't find the container with id 443872558dfe612d5317eefb9b1b1ce13a018b14230d0ae0b41793eb4ecc2238 Oct 31 01:21:08.838479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.838436 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:08.882015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.881731 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" event={"ID":"612909f8-0549-482d-bff0-8c961a462eae","Type":"ContainerStarted","Data":"443872558dfe612d5317eefb9b1b1ce13a018b14230d0ae0b41793eb4ecc2238"} Oct 31 01:21:08.883712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.883679 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" event={"ID":"5da9b719-1a5c-46b5-a49d-e0d3103c758e","Type":"ContainerStarted","Data":"4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069"} Oct 31 01:21:08.883851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.883732 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" event={"ID":"5da9b719-1a5c-46b5-a49d-e0d3103c758e","Type":"ContainerStarted","Data":"170c547ddb7292d8eab38d3ba2009c080efd52080afbbb830fdccc43c0768214"} Oct 31 01:21:08.883851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.883805 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:08.942420 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:08.940027 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" podStartSLOduration=1.940006299 podStartE2EDuration="1.940006299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:08.936964426 +0000 UTC m=+3686.635078190" watchObservedRunningTime="2026-10-31 01:21:08.940006299 +0000 UTC m=+3686.638120066" Oct 31 01:21:09.133737 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:09.133693 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:09.892037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:09.891991 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" event={"ID":"612909f8-0549-482d-bff0-8c961a462eae","Type":"ContainerStarted","Data":"e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473"} Oct 31 01:21:09.892555 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:09.892066 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:09.931957 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:09.931893 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" podStartSLOduration=1.931877987 podStartE2EDuration="1.931877987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:09.92887151 +0000 UTC m=+3687.626985275" watchObservedRunningTime="2026-10-31 01:21:09.931877987 +0000 UTC m=+3687.629991762" Oct 31 01:21:10.386560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:10.386517 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:10.539672 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:10.539628 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:10.681408 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:10.681309 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:12.701991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:12.698040 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:14.085192 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:14.085156 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:14.240026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:14.239974 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:14.384203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:14.384106 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:14.596367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:14.596328 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:14.736845 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:14.736754 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:15.250665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:15.250628 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:16.572396 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:16.572359 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:16.916379 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:16.916270 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:17.059052 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:17.059019 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:18.057662 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:18.057623 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:18.659859 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:18.659827 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:18.800117 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:18.800081 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:19.285081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:19.285045 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:19.895881 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:19.895850 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:21:20.279134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:20.279031 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:20.717031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:20.716974 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:20.964982 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:20.964954 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:21:22.408427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:22.408397 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:23.967326 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:23.967239 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:24.467340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:24.467299 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:24.635134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:24.635092 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:25.115597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:25.115564 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:26.560597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:26.560564 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:26.925194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:26.925159 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:27.081427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:27.081377 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:27.217499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:27.217391 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv"] Oct 31 01:21:27.217713 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:27.217680 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" containerID="cri-o://99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127" gracePeriod=30 Oct 31 01:21:27.299587 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:27.299543 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:28.813642 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:28.813591 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:30.211663 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:30.211623 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:30.736159 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:30.736120 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:30.880634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:30.880593 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:31.021081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:31.020982 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:31.401366 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:31.401329 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:31.915127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:31.915092 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:32.075609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.075573 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:32.210302 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.210219 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf"] Oct 31 01:21:32.210528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.210496 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" containerID="cri-o://f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40" gracePeriod=30 Oct 31 01:21:32.280124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.280091 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:32.521447 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.521422 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 01:21:32.565203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.565174 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istiod-ca-cert\") pod \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " Oct 31 01:21:32.565391 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.565215 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-socket\") pod \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " Oct 31 01:21:32.565391 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.565298 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-certs\") pod \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " Oct 31 01:21:32.565391 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.565352 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-token\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-token\") pod \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " Oct 31 01:21:32.565561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.565395 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-envoy\") pod \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " Oct 31 01:21:32.565561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.565501 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-podinfo\") pod \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " Oct 31 01:21:32.565561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.565556 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-credential-socket\") pod \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " Oct 31 01:21:32.565728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.565622 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-data\") pod \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " Oct 31 01:21:32.565728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.565661 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-kube-api-access-snpdk\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-kube-api-access-snpdk\") pod \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\" (UID: \"902de6f5-c680-450b-87ed-3c0ad7bd3e88\") " Oct 31 01:21:32.566331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.566234 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-certs" pod "902de6f5-c680-450b-87ed-3c0ad7bd3e88" (UID: "902de6f5-c680-450b-87ed-3c0ad7bd3e88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:32.566932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.566908 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:32.567200 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.567162 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-socket" pod "902de6f5-c680-450b-87ed-3c0ad7bd3e88" (UID: "902de6f5-c680-450b-87ed-3c0ad7bd3e88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:32.567200 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.567182 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-data" pod "902de6f5-c680-450b-87ed-3c0ad7bd3e88" (UID: "902de6f5-c680-450b-87ed-3c0ad7bd3e88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:32.567402 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.567352 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-credential-socket" pod "902de6f5-c680-450b-87ed-3c0ad7bd3e88" (UID: "902de6f5-c680-450b-87ed-3c0ad7bd3e88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:32.567520 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.567497 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istiod-ca-cert" pod "902de6f5-c680-450b-87ed-3c0ad7bd3e88" (UID: "902de6f5-c680-450b-87ed-3c0ad7bd3e88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:32.569564 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.569494 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-envoy" pod "902de6f5-c680-450b-87ed-3c0ad7bd3e88" (UID: "902de6f5-c680-450b-87ed-3c0ad7bd3e88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:32.569564 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.569542 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-token" pod "902de6f5-c680-450b-87ed-3c0ad7bd3e88" (UID: "902de6f5-c680-450b-87ed-3c0ad7bd3e88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:32.569734 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.569628 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-kube-api-access-snpdk" pod "902de6f5-c680-450b-87ed-3c0ad7bd3e88" (UID: "902de6f5-c680-450b-87ed-3c0ad7bd3e88"). InnerVolumeSpecName "kube-api-access-snpdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:32.571311 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.571276 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-podinfo" pod "902de6f5-c680-450b-87ed-3c0ad7bd3e88" (UID: "902de6f5-c680-450b-87ed-3c0ad7bd3e88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:21:32.667845 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.667812 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:32.667845 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.667838 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:32.667845 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.667847 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:32.668101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.667856 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:32.668101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.667865 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:32.668101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.667873 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:32.668101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.667881 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902de6f5-c680-450b-87ed-3c0ad7bd3e88-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:32.668101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.667890 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snpdk\" (UniqueName: \"kubernetes.io/projected/902de6f5-c680-450b-87ed-3c0ad7bd3e88-kube-api-access-snpdk\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:32.800861 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.800775 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:32.947545 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:32.947505 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:33.126997 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.126960 5978 generic.go:356] "Generic (PLEG): container finished" podID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerID="99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127" exitCode=0 Oct 31 01:21:33.127183 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.127019 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" Oct 31 01:21:33.127183 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.127048 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" event={"ID":"902de6f5-c680-450b-87ed-3c0ad7bd3e88","Type":"ContainerDied","Data":"99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127"} Oct 31 01:21:33.127183 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.127110 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv" event={"ID":"902de6f5-c680-450b-87ed-3c0ad7bd3e88","Type":"ContainerDied","Data":"dbdc104003a879f9d1ceb6aa7002c1cccde2ca3824cb9f7ee74bd5bc20088ca6"} Oct 31 01:21:33.127183 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.127132 5978 scope.go:122] "RemoveContainer" containerID="99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127" Oct 31 01:21:33.139267 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.139246 5978 scope.go:122] "RemoveContainer" containerID="99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127" Oct 31 01:21:33.139578 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:33.139558 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127\": container with ID starting with 99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127 not found: ID does not exist" containerID="99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127" Oct 31 01:21:33.139674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.139584 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127"} err="failed to get container status \"99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127\": rpc error: code = NotFound desc = could not find container \"99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127\": container with ID starting with 99d2c865ba33edb80603745656dd73fa27162e1bfd0137ec5d688ce1860dd127 not found: ID does not exist" Oct 31 01:21:33.172975 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.172944 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv"] Oct 31 01:21:33.182962 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.182928 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b024b5-openshift-default-75d4b8c46f-7fxkv"] Oct 31 01:21:33.185285 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:33.185254 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:21:33.408516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.408414 5978 patch_prober.go:29] interesting pod/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:21:33.408516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:33.408493 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:21:34.053474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:34.049241 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:34.427049 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:34.427012 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" path="/var/lib/kubelet/pods/902de6f5-c680-450b-87ed-3c0ad7bd3e88/volumes" Oct 31 01:21:34.755713 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:34.755618 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:36.420398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:36.420367 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:37.457286 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.457220 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:37.506310 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.506228 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 01:21:37.604643 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.604607 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:37.628504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.628461 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-socket\") pod \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " Oct 31 01:21:37.628684 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.628550 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-envoy\") pod \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " Oct 31 01:21:37.628684 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.628632 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istiod-ca-cert\") pod \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " Oct 31 01:21:37.628684 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.628665 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-token\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-token\") pod \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " Oct 31 01:21:37.628826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.628707 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-podinfo\") pod \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " Oct 31 01:21:37.628826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.628724 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-kube-api-access-nsf6c\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-kube-api-access-nsf6c\") pod \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " Oct 31 01:21:37.628826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.628757 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-credential-socket\") pod \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " Oct 31 01:21:37.628826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.628806 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-certs\") pod \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " Oct 31 01:21:37.628958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.628873 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-data\") pod \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\" (UID: \"f7d3430e-a53d-4202-acd3-bdc2a1faa943\") " Oct 31 01:21:37.629619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.629340 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-socket" pod "f7d3430e-a53d-4202-acd3-bdc2a1faa943" (UID: "f7d3430e-a53d-4202-acd3-bdc2a1faa943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:37.629751 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.629617 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-certs" pod "f7d3430e-a53d-4202-acd3-bdc2a1faa943" (UID: "f7d3430e-a53d-4202-acd3-bdc2a1faa943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:37.629751 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.629634 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istiod-ca-cert" pod "f7d3430e-a53d-4202-acd3-bdc2a1faa943" (UID: "f7d3430e-a53d-4202-acd3-bdc2a1faa943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:37.629881 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.629851 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-data" pod "f7d3430e-a53d-4202-acd3-bdc2a1faa943" (UID: "f7d3430e-a53d-4202-acd3-bdc2a1faa943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:37.630010 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.629987 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-credential-socket" pod "f7d3430e-a53d-4202-acd3-bdc2a1faa943" (UID: "f7d3430e-a53d-4202-acd3-bdc2a1faa943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:37.631398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.631376 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-envoy" pod "f7d3430e-a53d-4202-acd3-bdc2a1faa943" (UID: "f7d3430e-a53d-4202-acd3-bdc2a1faa943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:37.631515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.631492 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-podinfo" pod "f7d3430e-a53d-4202-acd3-bdc2a1faa943" (UID: "f7d3430e-a53d-4202-acd3-bdc2a1faa943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:21:37.631515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.631508 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-token" pod "f7d3430e-a53d-4202-acd3-bdc2a1faa943" (UID: "f7d3430e-a53d-4202-acd3-bdc2a1faa943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:37.631740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.631723 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-kube-api-access-nsf6c" pod "f7d3430e-a53d-4202-acd3-bdc2a1faa943" (UID: "f7d3430e-a53d-4202-acd3-bdc2a1faa943"). InnerVolumeSpecName "kube-api-access-nsf6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:37.730824 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.730788 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:37.730824 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.730825 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:37.731123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.730838 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:37.731123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.730852 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsf6c\" (UniqueName: \"kubernetes.io/projected/f7d3430e-a53d-4202-acd3-bdc2a1faa943-kube-api-access-nsf6c\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:37.731123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.730867 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:37.731123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.730880 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:37.731123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.730892 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:37.731123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.730904 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:37.731123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.730919 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d3430e-a53d-4202-acd3-bdc2a1faa943-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:21:37.842969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:37.842930 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:38.164148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.164060 5978 generic.go:356] "Generic (PLEG): container finished" podID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerID="f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40" exitCode=0 Oct 31 01:21:38.164148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.164118 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" Oct 31 01:21:38.164362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.164145 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" event={"ID":"f7d3430e-a53d-4202-acd3-bdc2a1faa943","Type":"ContainerDied","Data":"f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40"} Oct 31 01:21:38.164362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.164202 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf" event={"ID":"f7d3430e-a53d-4202-acd3-bdc2a1faa943","Type":"ContainerDied","Data":"489a35c98d7a80604ad318551707b1add0fe1bfbde2c43f2cd44f469d129a021"} Oct 31 01:21:38.164362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.164224 5978 scope.go:122] "RemoveContainer" containerID="f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40" Oct 31 01:21:38.176837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.176818 5978 scope.go:122] "RemoveContainer" containerID="8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589" Oct 31 01:21:38.182195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.182071 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:38.204130 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.204105 5978 scope.go:122] "RemoveContainer" containerID="f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40" Oct 31 01:21:38.204724 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:38.204687 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40\": container with ID starting with f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40 not found: ID does not exist" containerID="f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40" Oct 31 01:21:38.204875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.204732 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40"} err="failed to get container status \"f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40\": rpc error: code = NotFound desc = could not find container \"f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40\": container with ID starting with f90a11940df9e03fce5d1db191d4f5afe9936fd6a1f8867d41e2272d4d963f40 not found: ID does not exist" Oct 31 01:21:38.204875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.204754 5978 scope.go:122] "RemoveContainer" containerID="8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589" Oct 31 01:21:38.205143 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:21:38.205115 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589\": container with ID starting with 8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589 not found: ID does not exist" containerID="8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589" Oct 31 01:21:38.205249 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.205151 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589"} err="failed to get container status \"8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589\": rpc error: code = NotFound desc = could not find container \"8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589\": container with ID starting with 8183fe2749f90e114febbf7d43727c2d2ab9d5da85f99b0edb2c4ee3f7522589 not found: ID does not exist" Oct 31 01:21:38.211368 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.211348 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf"] Oct 31 01:21:38.214013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.213989 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-bcdb4e-openshift-default-5f88bc4687-x8zkf"] Oct 31 01:21:38.426147 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:38.426074 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" path="/var/lib/kubelet/pods/f7d3430e-a53d-4202-acd3-bdc2a1faa943/volumes" Oct 31 01:21:39.867635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:39.867599 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:41.972013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:41.971978 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:43.340342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:43.340304 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x"] Oct 31 01:21:43.512806 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:43.512771 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:43.656776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:43.656679 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:45.106157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:45.106118 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:45.486798 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:45.486701 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:45.632996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:45.632959 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:46.693093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:46.693061 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:48.298394 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:48.298348 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:48.341541 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:48.341446 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" podUID="770d1ffc-662a-4979-a708-ba009efd843e" containerName="payload-processing" containerID="cri-o://991f5d42b558bdf620178c373a80dc7643db6f04e2ea3e73f0e636e4377026d5" gracePeriod=25 Oct 31 01:21:48.855440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:48.855410 5978 scope.go:122] "RemoveContainer" containerID="ff4388c41f6e08e5e2dfe42731798a60187efca26fef5ac597d9288905010c6b" Oct 31 01:21:49.872772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:49.872735 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:50.439997 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:50.439962 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:51.574275 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:51.574239 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:51.806468 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:51.806419 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:53.305845 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:53.305810 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:55.387756 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:55.387718 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:56.792986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:56.792948 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" podUID="770d1ffc-662a-4979-a708-ba009efd843e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:21:56.888543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:56.888506 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:57.032482 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:57.030012 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:58.239444 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:58.239357 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:58.526693 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:58.526613 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:59.779672 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:59.779625 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr"] Oct 31 01:21:59.780367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:59.780317 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" podUID="4cd2a420-72b7-4763-be0f-888be3754b5b" containerName="maas-api" containerID="cri-o://c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce" gracePeriod=30 Oct 31 01:21:59.879209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:21:59.879159 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:00.239071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.239028 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 01:22:00.334098 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.334064 5978 generic.go:356] "Generic (PLEG): container finished" podID="4cd2a420-72b7-4763-be0f-888be3754b5b" containerID="c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce" exitCode=0 Oct 31 01:22:00.334275 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.334137 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" Oct 31 01:22:00.334275 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.334158 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" event={"ID":"4cd2a420-72b7-4763-be0f-888be3754b5b","Type":"ContainerDied","Data":"c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce"} Oct 31 01:22:00.334275 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.334224 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr" event={"ID":"4cd2a420-72b7-4763-be0f-888be3754b5b","Type":"ContainerDied","Data":"bf19765cc9ef7319b5d6c8508ec3f0fba720e10e09ef5d84fb8f79015545515c"} Oct 31 01:22:00.334275 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.334248 5978 scope.go:122] "RemoveContainer" containerID="c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce" Oct 31 01:22:00.351506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.351479 5978 scope.go:122] "RemoveContainer" containerID="c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce" Oct 31 01:22:00.351842 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:22:00.351813 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce\": container with ID starting with c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce not found: ID does not exist" containerID="c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce" Oct 31 01:22:00.351926 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.351846 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce"} err="failed to get container status \"c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce\": rpc error: code = NotFound desc = could not find container \"c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce\": container with ID starting with c10cff53b4a657e80d19c1d47591784b3b001738b19c73c8b676b04cdeec93ce not found: ID does not exist" Oct 31 01:22:00.363172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.357397 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd2a420-72b7-4763-be0f-888be3754b5b-kube-api-access-b8t4x\" (UniqueName: \"kubernetes.io/projected/4cd2a420-72b7-4763-be0f-888be3754b5b-kube-api-access-b8t4x\") pod \"4cd2a420-72b7-4763-be0f-888be3754b5b\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " Oct 31 01:22:00.363172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.357473 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls\") pod \"4cd2a420-72b7-4763-be0f-888be3754b5b\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " Oct 31 01:22:00.363172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.357525 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-metrics-certs\") pod \"4cd2a420-72b7-4763-be0f-888be3754b5b\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " Oct 31 01:22:00.363172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.357639 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd2a420-72b7-4763-be0f-888be3754b5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a420-72b7-4763-be0f-888be3754b5b-openshift-service-ca\") pod \"4cd2a420-72b7-4763-be0f-888be3754b5b\" (UID: \"4cd2a420-72b7-4763-be0f-888be3754b5b\") " Oct 31 01:22:00.363172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.358304 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd2a420-72b7-4763-be0f-888be3754b5b-openshift-service-ca" pod "4cd2a420-72b7-4763-be0f-888be3754b5b" (UID: "4cd2a420-72b7-4763-be0f-888be3754b5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:00.363172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.359985 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd2a420-72b7-4763-be0f-888be3754b5b-kube-api-access-b8t4x" pod "4cd2a420-72b7-4763-be0f-888be3754b5b" (UID: "4cd2a420-72b7-4763-be0f-888be3754b5b"). InnerVolumeSpecName "kube-api-access-b8t4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:00.363172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.360074 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-metrics-certs" pod "4cd2a420-72b7-4763-be0f-888be3754b5b" (UID: "4cd2a420-72b7-4763-be0f-888be3754b5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:00.363172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.360193 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls" pod "4cd2a420-72b7-4763-be0f-888be3754b5b" (UID: "4cd2a420-72b7-4763-be0f-888be3754b5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:00.452500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.452383 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:00.460010 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.459979 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2a420-72b7-4763-be0f-888be3754b5b-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:00.460133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.460016 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8t4x\" (UniqueName: \"kubernetes.io/projected/4cd2a420-72b7-4763-be0f-888be3754b5b-kube-api-access-b8t4x\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:00.460133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.460032 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:00.460133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.460045 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd2a420-72b7-4763-be0f-888be3754b5b-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:00.597829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.597789 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:00.684734 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.684698 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr"] Oct 31 01:22:00.690644 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.690617 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-714f8f-554454dc8c-s9srr"] Oct 31 01:22:00.749577 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:00.749498 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:01.117923 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:01.117882 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:02.352760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.352725 5978 generic.go:356] "Generic (PLEG): container finished" podID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerID="c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591" exitCode=0 Oct 31 01:22:02.353334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.352791 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" event={"ID":"b3feb663-8df2-41f4-acfa-4f883edb3740","Type":"ContainerDied","Data":"c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591"} Oct 31 01:22:02.353334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.353276 5978 scope.go:122] "RemoveContainer" containerID="c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591" Oct 31 01:22:02.355878 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.355826 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:02.356011 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.355923 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:02.356011 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.355964 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:02.356182 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.356163 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s"] Oct 31 01:22:02.356957 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.356938 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" Oct 31 01:22:02.357046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.356960 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" Oct 31 01:22:02.357046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.357000 5978 cpu_manager.go:414] "containerMap: removing container" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" Oct 31 01:22:02.357046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.357008 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" Oct 31 01:22:02.357046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.357020 5978 cpu_manager.go:414] "containerMap: removing container" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" Oct 31 01:22:02.357046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.357025 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" Oct 31 01:22:02.357354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.357059 5978 cpu_manager.go:414] "containerMap: removing container" podUID="4cd2a420-72b7-4763-be0f-888be3754b5b" containerName="maas-api" Oct 31 01:22:02.357354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.357071 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd2a420-72b7-4763-be0f-888be3754b5b" containerName="maas-api" Oct 31 01:22:02.357354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.357234 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" Oct 31 01:22:02.357354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.357244 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="902de6f5-c680-450b-87ed-3c0ad7bd3e88" containerName="istio-proxy" Oct 31 01:22:02.357354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.357254 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd2a420-72b7-4763-be0f-888be3754b5b" containerName="maas-api" Oct 31 01:22:02.359808 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.359790 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.390656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.390627 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s"] Oct 31 01:22:02.441425 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.441375 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:02.447579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.447539 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd2a420-72b7-4763-be0f-888be3754b5b" path="/var/lib/kubelet/pods/4cd2a420-72b7-4763-be0f-888be3754b5b/volumes" Oct 31 01:22:02.487671 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.487638 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.487869 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.487709 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.487869 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.487808 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2m86\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-kube-api-access-n2m86\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.488003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.487984 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.488068 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.488023 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.488129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.488090 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f68f517-8050-41a3-8dc3-690f0a9cb036-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.488184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.488140 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.488239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.488218 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.488612 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.488584 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590148 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f68f517-8050-41a3-8dc3-690f0a9cb036-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590188 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590244 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590281 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590367 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590414 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590705 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590447 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2m86\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-kube-api-access-n2m86\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590705 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590555 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590705 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590591 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590881 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590723 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.590881 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.590783 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.591043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.591005 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.591043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.591012 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f68f517-8050-41a3-8dc3-690f0a9cb036-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.591232 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.591209 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.593205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.593175 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.593339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.593234 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.612320 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.612249 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2m86\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-kube-api-access-n2m86\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.612608 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.612591 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.621341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.621309 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5"] Oct 31 01:22:02.659436 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.659399 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2"] Oct 31 01:22:02.682691 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.682657 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:02.725735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.725698 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:02.897286 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:02.897193 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:03.050104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.050078 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s"] Oct 31 01:22:03.052328 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:22:03.052292 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f68f517_8050_41a3_8dc3_690f0a9cb036.slice/crio-0486d0f77482f63c63b70b48fd03fee38d1e507a3a8fd6c15782abbaa2f4d882 WatchSource:0}: Error finding container 0486d0f77482f63c63b70b48fd03fee38d1e507a3a8fd6c15782abbaa2f4d882: Status 404 returned error can't find the container with id 0486d0f77482f63c63b70b48fd03fee38d1e507a3a8fd6c15782abbaa2f4d882 Oct 31 01:22:03.055359 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.055318 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:03.055498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.055395 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:03.055498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.055429 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:03.182550 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:22:03.182515 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:22:03.363332 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.363294 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" event={"ID":"0f68f517-8050-41a3-8dc3-690f0a9cb036","Type":"ContainerStarted","Data":"4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675"} Oct 31 01:22:03.363828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.363356 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" event={"ID":"0f68f517-8050-41a3-8dc3-690f0a9cb036","Type":"ContainerStarted","Data":"0486d0f77482f63c63b70b48fd03fee38d1e507a3a8fd6c15782abbaa2f4d882"} Oct 31 01:22:03.366177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.366149 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" event={"ID":"b3feb663-8df2-41f4-acfa-4f883edb3740","Type":"ContainerStarted","Data":"1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90"} Oct 31 01:22:03.416531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.411353 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" podStartSLOduration=1.411332996 podStartE2EDuration="1.411332996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:03.407055195 +0000 UTC m=+3741.105168997" watchObservedRunningTime="2026-10-31 01:22:03.411332996 +0000 UTC m=+3741.109446763" Oct 31 01:22:03.479140 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.479103 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:22:03.480639 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.480613 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:22:03.480767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.480662 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 01:22:03.660954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.660921 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:03.682842 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.682768 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:03.833477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:03.810879 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:04.479240 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:04.479197 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:22:04.479732 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:04.479255 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 01:22:04.534478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:04.528232 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:04.685186 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:04.685155 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 01:22:04.685359 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:04.685206 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 01:22:04.860480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:04.858554 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:22:05.032479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:05.032382 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:05.479884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:05.479851 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:22:05.480344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:05.480316 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 01:22:05.683800 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:05.683758 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 01:22:05.683993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:05.683818 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 01:22:06.479285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:06.479247 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:22:06.479523 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:06.479300 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 01:22:06.683946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:06.683905 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 01:22:06.684502 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:06.683979 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 01:22:06.803591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:06.803498 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:06.805182 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:06.804655 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" podUID="770d1ffc-662a-4979-a708-ba009efd843e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:22:07.479378 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:07.479343 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:22:07.479567 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:07.479395 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 01:22:07.630477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:07.622530 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" podUID="612909f8-0549-482d-bff0-8c961a462eae" containerName="payload-processing" containerID="cri-o://e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473" gracePeriod=25 Oct 31 01:22:07.661251 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:07.660543 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" podUID="5da9b719-1a5c-46b5-a49d-e0d3103c758e" containerName="payload-pre-processing" containerID="cri-o://4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069" gracePeriod=25 Oct 31 01:22:07.683549 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:07.683520 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 01:22:07.683688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:07.683576 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 01:22:08.001200 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.001055 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:08.063505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.027705 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:22:08.080480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.079979 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612909f8-0549-482d-bff0-8c961a462eae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612909f8-0549-482d-bff0-8c961a462eae-plugins-config-volume\") pod \"612909f8-0549-482d-bff0-8c961a462eae\" (UID: \"612909f8-0549-482d-bff0-8c961a462eae\") " Oct 31 01:22:08.080480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.080151 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612909f8-0549-482d-bff0-8c961a462eae-kube-api-access-jwp76\" (UniqueName: \"kubernetes.io/projected/612909f8-0549-482d-bff0-8c961a462eae-kube-api-access-jwp76\") pod \"612909f8-0549-482d-bff0-8c961a462eae\" (UID: \"612909f8-0549-482d-bff0-8c961a462eae\") " Oct 31 01:22:08.080743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.080599 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612909f8-0549-482d-bff0-8c961a462eae-plugins-config-volume" pod "612909f8-0549-482d-bff0-8c961a462eae" (UID: "612909f8-0549-482d-bff0-8c961a462eae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:08.081486 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.080828 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612909f8-0549-482d-bff0-8c961a462eae-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:08.083299 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.083257 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612909f8-0549-482d-bff0-8c961a462eae-kube-api-access-jwp76" pod "612909f8-0549-482d-bff0-8c961a462eae" (UID: "612909f8-0549-482d-bff0-8c961a462eae"). InnerVolumeSpecName "kube-api-access-jwp76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:08.173720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.173686 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:22:08.174572 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.174542 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:08.181227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.181207 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da9b719-1a5c-46b5-a49d-e0d3103c758e-kube-api-access-mrwh5\" (UniqueName: \"kubernetes.io/projected/5da9b719-1a5c-46b5-a49d-e0d3103c758e-kube-api-access-mrwh5\") pod \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\" (UID: \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\") " Oct 31 01:22:08.181373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.181359 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da9b719-1a5c-46b5-a49d-e0d3103c758e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9b719-1a5c-46b5-a49d-e0d3103c758e-plugins-config-volume\") pod \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\" (UID: \"5da9b719-1a5c-46b5-a49d-e0d3103c758e\") " Oct 31 01:22:08.181720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.181698 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da9b719-1a5c-46b5-a49d-e0d3103c758e-plugins-config-volume" pod "5da9b719-1a5c-46b5-a49d-e0d3103c758e" (UID: "5da9b719-1a5c-46b5-a49d-e0d3103c758e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:08.181893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.181880 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9b719-1a5c-46b5-a49d-e0d3103c758e-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:08.181964 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.181895 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwp76\" (UniqueName: \"kubernetes.io/projected/612909f8-0549-482d-bff0-8c961a462eae-kube-api-access-jwp76\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:08.183448 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.183427 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da9b719-1a5c-46b5-a49d-e0d3103c758e-kube-api-access-mrwh5" pod "5da9b719-1a5c-46b5-a49d-e0d3103c758e" (UID: "5da9b719-1a5c-46b5-a49d-e0d3103c758e"). InnerVolumeSpecName "kube-api-access-mrwh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:08.286658 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.285200 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrwh5\" (UniqueName: \"kubernetes.io/projected/5da9b719-1a5c-46b5-a49d-e0d3103c758e-kube-api-access-mrwh5\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:08.401053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.401009 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:08.402553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.402529 5978 generic.go:356] "Generic (PLEG): container finished" podID="770d1ffc-662a-4979-a708-ba009efd843e" containerID="991f5d42b558bdf620178c373a80dc7643db6f04e2ea3e73f0e636e4377026d5" exitCode=0 Oct 31 01:22:08.402683 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.402646 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" event={"ID":"770d1ffc-662a-4979-a708-ba009efd843e","Type":"ContainerDied","Data":"991f5d42b558bdf620178c373a80dc7643db6f04e2ea3e73f0e636e4377026d5"} Oct 31 01:22:08.404019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.403994 5978 generic.go:356] "Generic (PLEG): container finished" podID="5da9b719-1a5c-46b5-a49d-e0d3103c758e" containerID="4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069" exitCode=0 Oct 31 01:22:08.404124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.404100 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" Oct 31 01:22:08.404196 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.404126 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" event={"ID":"5da9b719-1a5c-46b5-a49d-e0d3103c758e","Type":"ContainerDied","Data":"4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069"} Oct 31 01:22:08.404196 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.404174 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2" event={"ID":"5da9b719-1a5c-46b5-a49d-e0d3103c758e","Type":"ContainerDied","Data":"170c547ddb7292d8eab38d3ba2009c080efd52080afbbb830fdccc43c0768214"} Oct 31 01:22:08.404299 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.404199 5978 scope.go:122] "RemoveContainer" containerID="4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069" Oct 31 01:22:08.419482 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.410588 5978 generic.go:356] "Generic (PLEG): container finished" podID="612909f8-0549-482d-bff0-8c961a462eae" containerID="e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473" exitCode=0 Oct 31 01:22:08.425879 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.425604 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" Oct 31 01:22:08.477479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.427620 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" event={"ID":"612909f8-0549-482d-bff0-8c961a462eae","Type":"ContainerDied","Data":"e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473"} Oct 31 01:22:08.477479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.427702 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5" event={"ID":"612909f8-0549-482d-bff0-8c961a462eae","Type":"ContainerDied","Data":"443872558dfe612d5317eefb9b1b1ce13a018b14230d0ae0b41793eb4ecc2238"} Oct 31 01:22:08.500411 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.482791 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 01:22:08.500411 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.482869 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 01:22:08.601474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.600992 5978 scope.go:122] "RemoveContainer" containerID="4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069" Oct 31 01:22:08.605221 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:22:08.605185 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069\": container with ID starting with 4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069 not found: ID does not exist" containerID="4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069" Oct 31 01:22:08.605385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.605232 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069"} err="failed to get container status \"4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069\": rpc error: code = NotFound desc = could not find container \"4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069\": container with ID starting with 4fac2670f228047b1163460da2c27f9a4730791b1d727664bafce171e8a1c069 not found: ID does not exist" Oct 31 01:22:08.605385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.605260 5978 scope.go:122] "RemoveContainer" containerID="e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473" Oct 31 01:22:08.661739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.656706 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2"] Oct 31 01:22:08.661739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.661077 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-59cccc7879-x87r2"] Oct 31 01:22:08.684090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.684043 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5"] Oct 31 01:22:08.684592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.684559 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 01:22:08.684814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.684633 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 01:22:08.689999 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.689967 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-7497dc5fdf-r6vv5"] Oct 31 01:22:08.695147 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.694989 5978 scope.go:122] "RemoveContainer" containerID="e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473" Oct 31 01:22:08.695427 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:22:08.695397 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473\": container with ID starting with e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473 not found: ID does not exist" containerID="e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473" Oct 31 01:22:08.695543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.695443 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473"} err="failed to get container status \"e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473\": rpc error: code = NotFound desc = could not find container \"e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473\": container with ID starting with e13182fa97d417ea0710fc31609adb03f5ecf13db35a6da5490ea3a3e36bf473 not found: ID does not exist" Oct 31 01:22:08.787317 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.786245 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:08.792875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.792848 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 01:22:08.905717 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.905682 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770d1ffc-662a-4979-a708-ba009efd843e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770d1ffc-662a-4979-a708-ba009efd843e-plugins-config-volume\") pod \"770d1ffc-662a-4979-a708-ba009efd843e\" (UID: \"770d1ffc-662a-4979-a708-ba009efd843e\") " Oct 31 01:22:08.905717 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.905722 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770d1ffc-662a-4979-a708-ba009efd843e-kube-api-access-nvzwx\" (UniqueName: \"kubernetes.io/projected/770d1ffc-662a-4979-a708-ba009efd843e-kube-api-access-nvzwx\") pod \"770d1ffc-662a-4979-a708-ba009efd843e\" (UID: \"770d1ffc-662a-4979-a708-ba009efd843e\") " Oct 31 01:22:08.906184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.906154 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770d1ffc-662a-4979-a708-ba009efd843e-plugins-config-volume" pod "770d1ffc-662a-4979-a708-ba009efd843e" (UID: "770d1ffc-662a-4979-a708-ba009efd843e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:08.906356 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.906342 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770d1ffc-662a-4979-a708-ba009efd843e-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:08.909482 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.909322 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770d1ffc-662a-4979-a708-ba009efd843e-kube-api-access-nvzwx" pod "770d1ffc-662a-4979-a708-ba009efd843e" (UID: "770d1ffc-662a-4979-a708-ba009efd843e"). InnerVolumeSpecName "kube-api-access-nvzwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:08.942963 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:08.942626 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:09.007139 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.007104 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvzwx\" (UniqueName: \"kubernetes.io/projected/770d1ffc-662a-4979-a708-ba009efd843e-kube-api-access-nvzwx\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:09.441795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.441768 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" Oct 31 01:22:09.441795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.441773 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x" event={"ID":"770d1ffc-662a-4979-a708-ba009efd843e","Type":"ContainerDied","Data":"4fb281c30609dfe62d843dd6cd07ddd703f019c721b2daba86861572ca9034ff"} Oct 31 01:22:09.442093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.441839 5978 scope.go:122] "RemoveContainer" containerID="991f5d42b558bdf620178c373a80dc7643db6f04e2ea3e73f0e636e4377026d5" Oct 31 01:22:09.484230 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.484200 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:22:09.485224 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.485202 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:22:09.489683 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.489661 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x"] Oct 31 01:22:09.496747 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.496671 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-714f8f-85559d789f-vhc4x"] Oct 31 01:22:09.782727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.782644 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:09.783035 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.783012 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:09.784180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:09.784156 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:22:10.425486 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:10.425437 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da9b719-1a5c-46b5-a49d-e0d3103c758e" path="/var/lib/kubelet/pods/5da9b719-1a5c-46b5-a49d-e0d3103c758e/volumes" Oct 31 01:22:10.425905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:10.425867 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612909f8-0549-482d-bff0-8c961a462eae" path="/var/lib/kubelet/pods/612909f8-0549-482d-bff0-8c961a462eae/volumes" Oct 31 01:22:10.426216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:10.426202 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770d1ffc-662a-4979-a708-ba009efd843e" path="/var/lib/kubelet/pods/770d1ffc-662a-4979-a708-ba009efd843e/volumes" Oct 31 01:22:10.618067 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:10.618015 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:10.923921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:10.923871 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:11.063651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:11.063617 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:11.219281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:11.219187 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv"] Oct 31 01:22:11.219488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:11.219443 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" containerID="cri-o://23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f" gracePeriod=30 Oct 31 01:22:11.280349 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:11.280313 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:12.378624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:12.377917 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:13.795579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:13.795542 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl"] Oct 31 01:22:13.936741 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:13.936705 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:14.092412 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:14.092377 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:15.760235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:15.760200 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:16.489921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.489859 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:22:16.498585 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.498551 5978 generic.go:356] "Generic (PLEG): container finished" podID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerID="23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f" exitCode=0 Oct 31 01:22:16.498731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.498606 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" Oct 31 01:22:16.498731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.498627 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" event={"ID":"6b232e00-9245-48ae-a016-2c0320dbdb02","Type":"ContainerDied","Data":"23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f"} Oct 31 01:22:16.498731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.498675 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv" event={"ID":"6b232e00-9245-48ae-a016-2c0320dbdb02","Type":"ContainerDied","Data":"343c9ee01dfd937c8cb76803c5f32b90c6b2989abff586221fbe51a8644eef46"} Oct 31 01:22:16.498731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.498703 5978 scope.go:122] "RemoveContainer" containerID="23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f" Oct 31 01:22:16.507318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507293 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-credential-socket\") pod \"6b232e00-9245-48ae-a016-2c0320dbdb02\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " Oct 31 01:22:16.507427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507333 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-kube-api-access-znr9f\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-kube-api-access-znr9f\") pod \"6b232e00-9245-48ae-a016-2c0320dbdb02\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " Oct 31 01:22:16.507427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507366 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-token\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-token\") pod \"6b232e00-9245-48ae-a016-2c0320dbdb02\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " Oct 31 01:22:16.507427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507406 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-envoy\") pod \"6b232e00-9245-48ae-a016-2c0320dbdb02\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " Oct 31 01:22:16.507626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507444 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-certs\") pod \"6b232e00-9245-48ae-a016-2c0320dbdb02\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " Oct 31 01:22:16.507626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507503 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-socket\") pod \"6b232e00-9245-48ae-a016-2c0320dbdb02\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " Oct 31 01:22:16.507746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507629 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-podinfo\") pod \"6b232e00-9245-48ae-a016-2c0320dbdb02\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " Oct 31 01:22:16.507746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507660 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b232e00-9245-48ae-a016-2c0320dbdb02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b232e00-9245-48ae-a016-2c0320dbdb02-istiod-ca-cert\") pod \"6b232e00-9245-48ae-a016-2c0320dbdb02\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " Oct 31 01:22:16.507746 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507727 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-data\") pod \"6b232e00-9245-48ae-a016-2c0320dbdb02\" (UID: \"6b232e00-9245-48ae-a016-2c0320dbdb02\") " Oct 31 01:22:16.510489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.507919 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-socket" pod "6b232e00-9245-48ae-a016-2c0320dbdb02" (UID: "6b232e00-9245-48ae-a016-2c0320dbdb02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.510489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.508199 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b232e00-9245-48ae-a016-2c0320dbdb02-istiod-ca-cert" pod "6b232e00-9245-48ae-a016-2c0320dbdb02" (UID: "6b232e00-9245-48ae-a016-2c0320dbdb02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:16.510489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.510112 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-envoy" pod "6b232e00-9245-48ae-a016-2c0320dbdb02" (UID: "6b232e00-9245-48ae-a016-2c0320dbdb02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.510489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.510218 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.510489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.510387 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-kube-api-access-znr9f" pod "6b232e00-9245-48ae-a016-2c0320dbdb02" (UID: "6b232e00-9245-48ae-a016-2c0320dbdb02"). InnerVolumeSpecName "kube-api-access-znr9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:16.510995 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.510515 5978 scope.go:122] "RemoveContainer" containerID="23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f" Oct 31 01:22:16.510995 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:22:16.510815 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f\": container with ID starting with 23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f not found: ID does not exist" containerID="23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f" Oct 31 01:22:16.510995 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.510850 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f"} err="failed to get container status \"23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f\": rpc error: code = NotFound desc = could not find container \"23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f\": container with ID starting with 23ce81b7c916557a157212e890879cb506b09bd0cb1bf570bbd217017163f97f not found: ID does not exist" Oct 31 01:22:16.510995 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.510932 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-podinfo" pod "6b232e00-9245-48ae-a016-2c0320dbdb02" (UID: "6b232e00-9245-48ae-a016-2c0320dbdb02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:22:16.510995 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.510941 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b232e00-9245-48ae-a016-2c0320dbdb02-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.512534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.511002 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-data" pod "6b232e00-9245-48ae-a016-2c0320dbdb02" (UID: "6b232e00-9245-48ae-a016-2c0320dbdb02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.512534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.511195 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-certs" pod "6b232e00-9245-48ae-a016-2c0320dbdb02" (UID: "6b232e00-9245-48ae-a016-2c0320dbdb02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.512534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.511223 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-credential-socket" pod "6b232e00-9245-48ae-a016-2c0320dbdb02" (UID: "6b232e00-9245-48ae-a016-2c0320dbdb02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:16.512534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.511259 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-token" pod "6b232e00-9245-48ae-a016-2c0320dbdb02" (UID: "6b232e00-9245-48ae-a016-2c0320dbdb02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:16.612102 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.612065 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.612102 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.612101 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.612316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.612116 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.612316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.612131 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znr9f\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-kube-api-access-znr9f\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.612316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.612145 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.612316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.612157 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.612316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.612170 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b232e00-9245-48ae-a016-2c0320dbdb02-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:16.841069 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.841036 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv"] Oct 31 01:22:16.850514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:16.850487 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4207cd-openshift-default-58cb5b4b55-5pgtv"] Oct 31 01:22:17.352032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:17.349959 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:18.426137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:18.426100 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" path="/var/lib/kubelet/pods/6b232e00-9245-48ae-a016-2c0320dbdb02/volumes" Oct 31 01:22:19.153905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:19.152754 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:19.542310 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:19.542229 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9"] Oct 31 01:22:19.575229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:19.575195 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w"] Oct 31 01:22:19.718830 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:19.718790 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:19.861942 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:19.861907 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:21.312966 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:21.308685 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:22.881990 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:22.881933 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:24.472867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:24.472823 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:24.543377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:24.543322 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" podUID="fc746f8f-fe26-4bd8-99da-cee03ed4f520" containerName="payload-processing" containerID="cri-o://1d0f51bc566dc0154c499157fbf39af2bb79b08a0d865053704fe26e1fb0351c" gracePeriod=25 Oct 31 01:22:24.575633 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:24.575578 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" podUID="e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" containerName="payload-pre-processing" containerID="cri-o://8ae7d6522695890cc122f54652f0a6d715101cacc2028e0a94c800c67a778ef4" gracePeriod=25 Oct 31 01:22:25.577319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.577276 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:25.750931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.750897 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:25.945052 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.944958 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms"] Oct 31 01:22:25.945768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945745 5978 cpu_manager.go:414] "containerMap: removing container" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" Oct 31 01:22:25.945768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945766 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" Oct 31 01:22:25.945947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945809 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5da9b719-1a5c-46b5-a49d-e0d3103c758e" containerName="payload-pre-processing" Oct 31 01:22:25.945947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945815 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da9b719-1a5c-46b5-a49d-e0d3103c758e" containerName="payload-pre-processing" Oct 31 01:22:25.945947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945826 5978 cpu_manager.go:414] "containerMap: removing container" podUID="612909f8-0549-482d-bff0-8c961a462eae" containerName="payload-processing" Oct 31 01:22:25.945947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945831 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612909f8-0549-482d-bff0-8c961a462eae" containerName="payload-processing" Oct 31 01:22:25.945947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945839 5978 cpu_manager.go:414] "containerMap: removing container" podUID="770d1ffc-662a-4979-a708-ba009efd843e" containerName="payload-processing" Oct 31 01:22:25.945947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945845 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770d1ffc-662a-4979-a708-ba009efd843e" containerName="payload-processing" Oct 31 01:22:25.946234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945957 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d3430e-a53d-4202-acd3-bdc2a1faa943" containerName="istio-proxy" Oct 31 01:22:25.946234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945980 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da9b719-1a5c-46b5-a49d-e0d3103c758e" containerName="payload-pre-processing" Oct 31 01:22:25.946234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945988 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="612909f8-0549-482d-bff0-8c961a462eae" containerName="payload-processing" Oct 31 01:22:25.946234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.945994 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="770d1ffc-662a-4979-a708-ba009efd843e" containerName="payload-processing" Oct 31 01:22:25.946234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.946006 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b232e00-9245-48ae-a016-2c0320dbdb02" containerName="istio-proxy" Oct 31 01:22:25.950109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.950079 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:25.963819 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.963795 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8f0029\"" Oct 31 01:22:25.973395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.973365 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8f0029-dockercfg-pnsnm\"" Oct 31 01:22:25.983489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:25.983441 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms"] Oct 31 01:22:26.045160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.045118 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x2hj\" (UniqueName: \"kubernetes.io/projected/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-kube-api-access-6x2hj\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms\" (UID: \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:26.045399 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.045226 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms\" (UID: \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:26.120977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.120931 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn"] Oct 31 01:22:26.125786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.125760 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:26.146201 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.146169 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn"] Oct 31 01:22:26.146840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.146808 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x2hj\" (UniqueName: \"kubernetes.io/projected/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-kube-api-access-6x2hj\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms\" (UID: \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:26.146981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.146882 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms\" (UID: \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:26.147801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.147775 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms\" (UID: \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:26.168761 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.168721 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x2hj\" (UniqueName: \"kubernetes.io/projected/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-kube-api-access-6x2hj\") pod \"payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms\" (UID: \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:26.248403 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.248315 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9kp\" (UniqueName: \"kubernetes.io/projected/5e5f0fde-ba78-4c22-82fd-35547be085d1-kube-api-access-xk9kp\") pod \"payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn\" (UID: \"5e5f0fde-ba78-4c22-82fd-35547be085d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:26.248403 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.248394 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5f0fde-ba78-4c22-82fd-35547be085d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn\" (UID: \"5e5f0fde-ba78-4c22-82fd-35547be085d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:26.265019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.264987 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:26.349258 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.349222 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9kp\" (UniqueName: \"kubernetes.io/projected/5e5f0fde-ba78-4c22-82fd-35547be085d1-kube-api-access-xk9kp\") pod \"payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn\" (UID: \"5e5f0fde-ba78-4c22-82fd-35547be085d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:26.349431 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.349296 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5f0fde-ba78-4c22-82fd-35547be085d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn\" (UID: \"5e5f0fde-ba78-4c22-82fd-35547be085d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:26.350148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.350126 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5f0fde-ba78-4c22-82fd-35547be085d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn\" (UID: \"5e5f0fde-ba78-4c22-82fd-35547be085d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:26.370763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.370721 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9kp\" (UniqueName: \"kubernetes.io/projected/5e5f0fde-ba78-4c22-82fd-35547be085d1-kube-api-access-xk9kp\") pod \"payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn\" (UID: \"5e5f0fde-ba78-4c22-82fd-35547be085d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:26.440174 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.440136 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:26.564647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.564533 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms"] Oct 31 01:22:26.571410 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:22:26.571374 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab46bd84_b8e6_4b9e_a5eb_56ec944e26ea.slice/crio-f7b0de129b7d838a482029105c58c8adc320a41d3fe8703393fbae97b7bc875b WatchSource:0}: Error finding container f7b0de129b7d838a482029105c58c8adc320a41d3fe8703393fbae97b7bc875b: Status 404 returned error can't find the container with id f7b0de129b7d838a482029105c58c8adc320a41d3fe8703393fbae97b7bc875b Oct 31 01:22:26.676500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.676434 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:26.756798 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:26.756765 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn"] Oct 31 01:22:26.756967 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:22:26.756880 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5f0fde_ba78_4c22_82fd_35547be085d1.slice/crio-e7a44d44488e4d6af44422055aab18338ed994b166bdbbc3eb47e11e8b50e5b9 WatchSource:0}: Error finding container e7a44d44488e4d6af44422055aab18338ed994b166bdbbc3eb47e11e8b50e5b9: Status 404 returned error can't find the container with id e7a44d44488e4d6af44422055aab18338ed994b166bdbbc3eb47e11e8b50e5b9 Oct 31 01:22:27.579074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:27.579039 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" event={"ID":"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea","Type":"ContainerStarted","Data":"c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314"} Oct 31 01:22:27.579269 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:27.579105 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:27.579269 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:27.579128 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" event={"ID":"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea","Type":"ContainerStarted","Data":"f7b0de129b7d838a482029105c58c8adc320a41d3fe8703393fbae97b7bc875b"} Oct 31 01:22:27.580735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:27.580712 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" event={"ID":"5e5f0fde-ba78-4c22-82fd-35547be085d1","Type":"ContainerStarted","Data":"e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8"} Oct 31 01:22:27.580838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:27.580753 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" event={"ID":"5e5f0fde-ba78-4c22-82fd-35547be085d1","Type":"ContainerStarted","Data":"e7a44d44488e4d6af44422055aab18338ed994b166bdbbc3eb47e11e8b50e5b9"} Oct 31 01:22:27.580838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:27.580778 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:27.616890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:27.616824 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" podStartSLOduration=2.616804222 podStartE2EDuration="2.616804222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:27.615373368 +0000 UTC m=+3765.313487132" watchObservedRunningTime="2026-10-31 01:22:27.616804222 +0000 UTC m=+3765.314917993" Oct 31 01:22:27.653090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:27.653023 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" podStartSLOduration=1.653004787 podStartE2EDuration="1.653004787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:27.652331315 +0000 UTC m=+3765.350445082" watchObservedRunningTime="2026-10-31 01:22:27.653004787 +0000 UTC m=+3765.351118557" Oct 31 01:22:28.045213 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:28.045117 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:28.582505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:28.582424 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:28.727466 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:28.727412 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:28.817180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:28.817142 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" containerID="cri-o://ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7" gracePeriod=45 Oct 31 01:22:28.893407 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:22:28.893371 5978 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod501ba3a7_cbea_4081_bf2f_e19c6da9d390.slice/crio-conmon-ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:22:28.955896 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:28.955843 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:28.984044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:28.983996 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Oct 31 01:22:29.101610 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.101543 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 01:22:29.190093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.190059 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tmp-dir\") pod \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " Oct 31 01:22:29.190315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.190120 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-model-cache\") pod \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " Oct 31 01:22:29.190315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.190230 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501ba3a7-cbea-4081-bf2f-e19c6da9d390-kube-api-access-zx54d\" (UniqueName: \"kubernetes.io/projected/501ba3a7-cbea-4081-bf2f-e19c6da9d390-kube-api-access-zx54d\") pod \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " Oct 31 01:22:29.190315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.190291 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tls-certs\" (UniqueName: \"kubernetes.io/secret/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tls-certs\") pod \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " Oct 31 01:22:29.190517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.190333 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-dshm\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-dshm\") pod \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " Oct 31 01:22:29.190517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.190438 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-model-cache" pod "501ba3a7-cbea-4081-bf2f-e19c6da9d390" (UID: "501ba3a7-cbea-4081-bf2f-e19c6da9d390"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.190517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.190448 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-home\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-home\") pod \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\" (UID: \"501ba3a7-cbea-4081-bf2f-e19c6da9d390\") " Oct 31 01:22:29.190837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.190811 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tmp-dir" pod "501ba3a7-cbea-4081-bf2f-e19c6da9d390" (UID: "501ba3a7-cbea-4081-bf2f-e19c6da9d390"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.191090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.191064 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.191090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.191088 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.191335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.191305 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-home" pod "501ba3a7-cbea-4081-bf2f-e19c6da9d390" (UID: "501ba3a7-cbea-4081-bf2f-e19c6da9d390"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.192843 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.192815 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-dshm" pod "501ba3a7-cbea-4081-bf2f-e19c6da9d390" (UID: "501ba3a7-cbea-4081-bf2f-e19c6da9d390"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:29.192966 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.192936 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tls-certs" pod "501ba3a7-cbea-4081-bf2f-e19c6da9d390" (UID: "501ba3a7-cbea-4081-bf2f-e19c6da9d390"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:29.193046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.193027 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501ba3a7-cbea-4081-bf2f-e19c6da9d390-kube-api-access-zx54d" pod "501ba3a7-cbea-4081-bf2f-e19c6da9d390" (UID: "501ba3a7-cbea-4081-bf2f-e19c6da9d390"). InnerVolumeSpecName "kube-api-access-zx54d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:29.242218 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.242180 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:29.292076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.292041 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx54d\" (UniqueName: \"kubernetes.io/projected/501ba3a7-cbea-4081-bf2f-e19c6da9d390-kube-api-access-zx54d\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.292076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.292081 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501ba3a7-cbea-4081-bf2f-e19c6da9d390-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.292351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.292098 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.292351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.292109 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501ba3a7-cbea-4081-bf2f-e19c6da9d390-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:29.598228 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.598195 5978 generic.go:356] "Generic (PLEG): container finished" podID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerID="ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7" exitCode=0 Oct 31 01:22:29.598464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.598241 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" Oct 31 01:22:29.598464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.598286 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" event={"ID":"501ba3a7-cbea-4081-bf2f-e19c6da9d390","Type":"ContainerDied","Data":"ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7"} Oct 31 01:22:29.598464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.598332 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl" event={"ID":"501ba3a7-cbea-4081-bf2f-e19c6da9d390","Type":"ContainerDied","Data":"e239b6a73be7a157a3f1b65e929348cf1866b4f4a14dda48116e6a66db451b3e"} Oct 31 01:22:29.598464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.598355 5978 scope.go:122] "RemoveContainer" containerID="ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7" Oct 31 01:22:29.609804 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.609783 5978 scope.go:122] "RemoveContainer" containerID="ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7" Oct 31 01:22:29.610130 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:22:29.610099 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7\": container with ID starting with ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7 not found: ID does not exist" containerID="ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7" Oct 31 01:22:29.610215 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.610143 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7"} err="failed to get container status \"ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7\": rpc error: code = NotFound desc = could not find container \"ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7\": container with ID starting with ba57ae919f6d263c2d7c6f01ddf93624efe96d06424b9c9f6e1af123d9e058f7 not found: ID does not exist" Oct 31 01:22:29.643470 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.643423 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl"] Oct 31 01:22:29.652784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.652751 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-be301b/premium-w6-be301b-kserve-77b87789f9-mbgnl"] Oct 31 01:22:29.758383 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.758344 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" podUID="e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:22:29.760638 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:29.760606 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" podUID="fc746f8f-fe26-4bd8-99da-cee03ed4f520" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:22:30.426374 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:30.426339 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" path="/var/lib/kubelet/pods/501ba3a7-cbea-4081-bf2f-e19c6da9d390/volumes" Oct 31 01:22:30.984965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:30.984925 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:32.711246 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:32.711191 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:33.181126 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:22:33.181091 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:22:34.308944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:34.308909 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:35.606739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:35.606696 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:35.746249 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:35.746214 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:35.897170 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:35.897080 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:36.055671 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:36.055633 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:37.592342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:37.592299 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:37.835723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:37.835669 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:38.592282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:38.592243 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:22:38.592529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:38.592325 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:22:39.632343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:39.629550 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:39.758760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:39.758723 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" podUID="fc746f8f-fe26-4bd8-99da-cee03ed4f520" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:22:39.758977 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:39.758753 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" podUID="e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:22:41.128693 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:41.128651 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:41.287979 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:41.287924 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:41.441119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:41.441034 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:41.762830 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:41.762739 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:41.917579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:41.917547 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:42.380223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:42.380190 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:43.652920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:43.652886 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:44.715025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.714988 5978 generic.go:356] "Generic (PLEG): container finished" podID="fc746f8f-fe26-4bd8-99da-cee03ed4f520" containerID="1d0f51bc566dc0154c499157fbf39af2bb79b08a0d865053704fe26e1fb0351c" exitCode=0 Oct 31 01:22:44.715557 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.715145 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" event={"ID":"fc746f8f-fe26-4bd8-99da-cee03ed4f520","Type":"ContainerDied","Data":"1d0f51bc566dc0154c499157fbf39af2bb79b08a0d865053704fe26e1fb0351c"} Oct 31 01:22:44.716975 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.716946 5978 generic.go:356] "Generic (PLEG): container finished" podID="e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" containerID="8ae7d6522695890cc122f54652f0a6d715101cacc2028e0a94c800c67a778ef4" exitCode=0 Oct 31 01:22:44.717128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.717049 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" event={"ID":"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4","Type":"ContainerDied","Data":"8ae7d6522695890cc122f54652f0a6d715101cacc2028e0a94c800c67a778ef4"} Oct 31 01:22:44.861141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.861112 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 01:22:44.902958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.902929 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 01:22:44.998305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.998270 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc746f8f-fe26-4bd8-99da-cee03ed4f520-kube-api-access-tbbzn\" (UniqueName: \"kubernetes.io/projected/fc746f8f-fe26-4bd8-99da-cee03ed4f520-kube-api-access-tbbzn\") pod \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\" (UID: \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\") " Oct 31 01:22:44.998496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.998354 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc746f8f-fe26-4bd8-99da-cee03ed4f520-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc746f8f-fe26-4bd8-99da-cee03ed4f520-plugins-config-volume\") pod \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\" (UID: \"fc746f8f-fe26-4bd8-99da-cee03ed4f520\") " Oct 31 01:22:44.998496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.998392 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-kube-api-access-pdprp\" (UniqueName: \"kubernetes.io/projected/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-kube-api-access-pdprp\") pod \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\" (UID: \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\") " Oct 31 01:22:44.998496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.998430 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-plugins-config-volume\") pod \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\" (UID: \"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4\") " Oct 31 01:22:44.998846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.998814 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc746f8f-fe26-4bd8-99da-cee03ed4f520-plugins-config-volume" pod "fc746f8f-fe26-4bd8-99da-cee03ed4f520" (UID: "fc746f8f-fe26-4bd8-99da-cee03ed4f520"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:44.998846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.998830 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-plugins-config-volume" pod "e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" (UID: "e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:44.999044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.999029 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc746f8f-fe26-4bd8-99da-cee03ed4f520-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:44.999090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:44.999049 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:45.000704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.000682 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc746f8f-fe26-4bd8-99da-cee03ed4f520-kube-api-access-tbbzn" pod "fc746f8f-fe26-4bd8-99da-cee03ed4f520" (UID: "fc746f8f-fe26-4bd8-99da-cee03ed4f520"). InnerVolumeSpecName "kube-api-access-tbbzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:45.000814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.000769 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-kube-api-access-pdprp" pod "e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" (UID: "e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4"). InnerVolumeSpecName "kube-api-access-pdprp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:45.100051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.100013 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbzn\" (UniqueName: \"kubernetes.io/projected/fc746f8f-fe26-4bd8-99da-cee03ed4f520-kube-api-access-tbbzn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:45.100051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.100047 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdprp\" (UniqueName: \"kubernetes.io/projected/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4-kube-api-access-pdprp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:45.324506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.324422 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:45.727963 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.727927 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" Oct 31 01:22:45.727963 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.727941 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w" event={"ID":"e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4","Type":"ContainerDied","Data":"ef587facbc1e9fe9625aa0c1c6a5cce3b01c0d0b0bc120b743dd7810377cbe43"} Oct 31 01:22:45.728505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.728008 5978 scope.go:122] "RemoveContainer" containerID="8ae7d6522695890cc122f54652f0a6d715101cacc2028e0a94c800c67a778ef4" Oct 31 01:22:45.729630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.729609 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" event={"ID":"fc746f8f-fe26-4bd8-99da-cee03ed4f520","Type":"ContainerDied","Data":"87ecbed0fbeddf1171a8ec3a10913d51d0b45cd601063ab396fb608707b9c62c"} Oct 31 01:22:45.729733 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.729637 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9" Oct 31 01:22:45.742324 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.742300 5978 scope.go:122] "RemoveContainer" containerID="1d0f51bc566dc0154c499157fbf39af2bb79b08a0d865053704fe26e1fb0351c" Oct 31 01:22:45.772856 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.772808 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w"] Oct 31 01:22:45.781980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.781944 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be301b-57cdb5dfc9-p264w"] Oct 31 01:22:45.816980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.816949 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9"] Oct 31 01:22:45.820632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:45.820589 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be301b-77c7dbd46f-wfqh9"] Oct 31 01:22:46.425126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:46.425090 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" path="/var/lib/kubelet/pods/e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4/volumes" Oct 31 01:22:46.425556 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:46.425537 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc746f8f-fe26-4bd8-99da-cee03ed4f520" path="/var/lib/kubelet/pods/fc746f8f-fe26-4bd8-99da-cee03ed4f520/volumes" Oct 31 01:22:47.226155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.226117 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:47.430615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.430580 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:47.528446 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.528357 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc"] Oct 31 01:22:47.529053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.529028 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" containerName="payload-pre-processing" Oct 31 01:22:47.529053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.529052 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" containerName="payload-pre-processing" Oct 31 01:22:47.529256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.529077 5978 cpu_manager.go:414] "containerMap: removing container" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" Oct 31 01:22:47.529256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.529082 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" Oct 31 01:22:47.529256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.529101 5978 cpu_manager.go:414] "containerMap: removing container" podUID="fc746f8f-fe26-4bd8-99da-cee03ed4f520" containerName="payload-processing" Oct 31 01:22:47.529256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.529106 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc746f8f-fe26-4bd8-99da-cee03ed4f520" containerName="payload-processing" Oct 31 01:22:47.529256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.529194 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ef4823-0af5-4fa5-a498-9eab4a8fc4c4" containerName="payload-pre-processing" Oct 31 01:22:47.529256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.529207 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="501ba3a7-cbea-4081-bf2f-e19c6da9d390" containerName="main" Oct 31 01:22:47.529256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.529219 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc746f8f-fe26-4bd8-99da-cee03ed4f520" containerName="payload-processing" Oct 31 01:22:47.536958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.536915 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.560527 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.560490 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc"] Oct 31 01:22:47.585767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.585732 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:47.650003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.649964 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.650003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.650011 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.650285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.650039 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.650285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.650157 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.650285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.650196 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnhvk\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-kube-api-access-jnhvk\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.650285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.650225 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.650535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.650336 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.650535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.650385 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.650535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.650490 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.751665 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.751753 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.751806 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnhvk\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-kube-api-access-jnhvk\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.751845 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.751890 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.751918 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.751958 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.752132 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.752167 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.753364 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-data\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.754021 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.754236 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-credential-socket\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.754443 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-certs\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.760854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.759366 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istiod-ca-cert\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.765163 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.765131 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-envoy\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.779408 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.779291 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-token\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.782187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.782140 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnhvk\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-kube-api-access-jnhvk\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.782800 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.782771 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-podinfo\") pod \"e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:47.873883 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:47.873836 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:48.190666 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:48.190439 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc"] Oct 31 01:22:48.192546 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:22:48.192515 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ed5649_6ad5_4423_8784_0e2fc1ed64bc.slice/crio-3dc77fc5d1878efe071af289bbf07e67130c993f2c4cae704f09f96fcf4f50b3 WatchSource:0}: Error finding container 3dc77fc5d1878efe071af289bbf07e67130c993f2c4cae704f09f96fcf4f50b3: Status 404 returned error can't find the container with id 3dc77fc5d1878efe071af289bbf07e67130c993f2c4cae704f09f96fcf4f50b3 Oct 31 01:22:48.195247 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:48.195208 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:48.195374 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:48.195298 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:48.195374 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:48.195347 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:22:48.770293 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:48.770253 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" event={"ID":"71ed5649-6ad5-4423-8784-0e2fc1ed64bc","Type":"ContainerStarted","Data":"fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee"} Oct 31 01:22:48.770719 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:48.770321 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" event={"ID":"71ed5649-6ad5-4423-8784-0e2fc1ed64bc","Type":"ContainerStarted","Data":"3dc77fc5d1878efe071af289bbf07e67130c993f2c4cae704f09f96fcf4f50b3"} Oct 31 01:22:48.874378 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:48.874347 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:49.236510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:49.236100 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:49.384872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:49.384835 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:49.605306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:49.605270 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:49.876152 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:49.876073 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 01:22:49.876152 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:49.876137 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 01:22:50.701626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:50.701587 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:50.875269 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:50.875232 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 01:22:50.875509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:50.875288 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 01:22:51.428082 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:51.428043 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:51.588897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:51.588863 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:51.875148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:51.875108 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 01:22:51.875376 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:51.875169 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 01:22:51.965477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:51.955557 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:52.874949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:52.874914 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 01:22:52.875401 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:52.874974 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 01:22:53.680517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:53.680478 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:53.874982 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:53.874899 5978 patch_prober.go:29] interesting pod/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 31 01:22:53.874982 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:53.874960 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 31 01:22:53.892146 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:53.891374 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:54.879339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:54.879309 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:54.879831 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:54.879820 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:54.880628 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:54.880600 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:22:54.939936 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:54.939862 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" podStartSLOduration=7.939839849 podStartE2EDuration="7.939839849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:48.831836642 +0000 UTC m=+3786.529950408" watchObservedRunningTime="2026-10-31 01:22:54.939839849 +0000 UTC m=+3792.637953615" Oct 31 01:22:55.304720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.304619 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w"] Oct 31 01:22:55.305521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.305213 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" podUID="41d84e9d-39fe-400a-9981-f4588885fa53" containerName="maas-api" containerID="cri-o://bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf" gracePeriod=30 Oct 31 01:22:55.434606 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.434560 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:55.625296 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.625261 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:22:55.689697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.681044 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d84e9d-39fe-400a-9981-f4588885fa53-kube-api-access-ls5r7\" (UniqueName: \"kubernetes.io/projected/41d84e9d-39fe-400a-9981-f4588885fa53-kube-api-access-ls5r7\") pod \"41d84e9d-39fe-400a-9981-f4588885fa53\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " Oct 31 01:22:55.689697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.681204 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-metrics-certs\") pod \"41d84e9d-39fe-400a-9981-f4588885fa53\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " Oct 31 01:22:55.689697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.681332 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls\") pod \"41d84e9d-39fe-400a-9981-f4588885fa53\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " Oct 31 01:22:55.689697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.681389 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d84e9d-39fe-400a-9981-f4588885fa53-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d84e9d-39fe-400a-9981-f4588885fa53-openshift-service-ca\") pod \"41d84e9d-39fe-400a-9981-f4588885fa53\" (UID: \"41d84e9d-39fe-400a-9981-f4588885fa53\") " Oct 31 01:22:55.698177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.690077 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d84e9d-39fe-400a-9981-f4588885fa53-openshift-service-ca" pod "41d84e9d-39fe-400a-9981-f4588885fa53" (UID: "41d84e9d-39fe-400a-9981-f4588885fa53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:55.727778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.709625 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls" pod "41d84e9d-39fe-400a-9981-f4588885fa53" (UID: "41d84e9d-39fe-400a-9981-f4588885fa53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:55.763472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.761771 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d84e9d-39fe-400a-9981-f4588885fa53-kube-api-access-ls5r7" pod "41d84e9d-39fe-400a-9981-f4588885fa53" (UID: "41d84e9d-39fe-400a-9981-f4588885fa53"). InnerVolumeSpecName "kube-api-access-ls5r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:55.765120 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.763694 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-metrics-certs" pod "41d84e9d-39fe-400a-9981-f4588885fa53" (UID: "41d84e9d-39fe-400a-9981-f4588885fa53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:55.784062 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.784020 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:55.784062 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.784063 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d84e9d-39fe-400a-9981-f4588885fa53-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:55.784351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.784078 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls5r7\" (UniqueName: \"kubernetes.io/projected/41d84e9d-39fe-400a-9981-f4588885fa53-kube-api-access-ls5r7\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:55.784351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.784092 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d84e9d-39fe-400a-9981-f4588885fa53-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:22:55.816675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.816637 5978 generic.go:356] "Generic (PLEG): container finished" podID="41d84e9d-39fe-400a-9981-f4588885fa53" containerID="bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf" exitCode=0 Oct 31 01:22:55.816863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.816695 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" Oct 31 01:22:55.816863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.816724 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" event={"ID":"41d84e9d-39fe-400a-9981-f4588885fa53","Type":"ContainerDied","Data":"bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf"} Oct 31 01:22:55.816863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.816797 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w" event={"ID":"41d84e9d-39fe-400a-9981-f4588885fa53","Type":"ContainerDied","Data":"dc644444c8a3578573384053dec4e2997dcf0db5b542d36c154010e287fabb8a"} Oct 31 01:22:55.816863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.816829 5978 scope.go:122] "RemoveContainer" containerID="bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf" Oct 31 01:22:55.828826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.828792 5978 scope.go:122] "RemoveContainer" containerID="bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf" Oct 31 01:22:55.829201 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:22:55.829178 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf\": container with ID starting with bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf not found: ID does not exist" containerID="bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf" Oct 31 01:22:55.829270 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.829212 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf"} err="failed to get container status \"bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf\": rpc error: code = NotFound desc = could not find container \"bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf\": container with ID starting with bdd864b9947a6ccf27cfe686056c289e1e25ecf1d67a536a302aab81cad46ccf not found: ID does not exist" Oct 31 01:22:55.875604 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.875509 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w"] Oct 31 01:22:55.878168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.878139 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ada0c1-8696b55df4-lnb4w"] Oct 31 01:22:55.891578 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:55.891549 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:56.084284 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:56.084249 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:56.265207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:56.265120 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:56.426636 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:56.426604 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d84e9d-39fe-400a-9981-f4588885fa53" path="/var/lib/kubelet/pods/41d84e9d-39fe-400a-9981-f4588885fa53/volumes" Oct 31 01:22:56.586380 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:56.586346 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:57.727786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:57.727738 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:58.236821 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:58.236785 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:58.400004 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:58.399951 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:58.892219 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:22:58.892183 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:00.173283 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:00.173243 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:00.596104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:00.596064 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:00.780872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:00.780839 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:00.942894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:00.942808 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:01.888194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:01.888160 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:02.377469 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:02.377425 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd"] Oct 31 01:23:02.377697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:02.377672 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" podUID="67469c9e-8e6c-4a36-a4e8-d85a1f010121" containerName="discovery" containerID="cri-o://c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb" gracePeriod=30 Oct 31 01:23:03.182521 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:03.182484 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:23:03.451921 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.451744 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 01:23:03.501610 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.501576 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-dns-cert\") pod \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " Oct 31 01:23:03.501777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.501634 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67469c9e-8e6c-4a36-a4e8-d85a1f010121-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67469c9e-8e6c-4a36-a4e8-d85a1f010121-local-certs\") pod \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " Oct 31 01:23:03.501777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.501658 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-kube-api-access-47wzm\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-kube-api-access-47wzm\") pod \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " Oct 31 01:23:03.501777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.501683 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-token\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-token\") pod \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " Oct 31 01:23:03.501777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.501717 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-kubeconfig\") pod \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " Oct 31 01:23:03.501777 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.501768 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-ca-configmap\") pod \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " Oct 31 01:23:03.502013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.501883 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-cacerts\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-cacerts\") pod \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\" (UID: \"67469c9e-8e6c-4a36-a4e8-d85a1f010121\") " Oct 31 01:23:03.503112 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.503072 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-ca-configmap" pod "67469c9e-8e6c-4a36-a4e8-d85a1f010121" (UID: "67469c9e-8e6c-4a36-a4e8-d85a1f010121"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:03.505089 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.505042 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67469c9e-8e6c-4a36-a4e8-d85a1f010121-local-certs" pod "67469c9e-8e6c-4a36-a4e8-d85a1f010121" (UID: "67469c9e-8e6c-4a36-a4e8-d85a1f010121"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:03.505210 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.505153 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-cacerts" pod "67469c9e-8e6c-4a36-a4e8-d85a1f010121" (UID: "67469c9e-8e6c-4a36-a4e8-d85a1f010121"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:03.505401 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.505378 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-dns-cert" pod "67469c9e-8e6c-4a36-a4e8-d85a1f010121" (UID: "67469c9e-8e6c-4a36-a4e8-d85a1f010121"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:03.505716 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.505695 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-kubeconfig" pod "67469c9e-8e6c-4a36-a4e8-d85a1f010121" (UID: "67469c9e-8e6c-4a36-a4e8-d85a1f010121"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:03.505795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.505733 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-kube-api-access-47wzm" pod "67469c9e-8e6c-4a36-a4e8-d85a1f010121" (UID: "67469c9e-8e6c-4a36-a4e8-d85a1f010121"). InnerVolumeSpecName "kube-api-access-47wzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:03.505795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.505771 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-token" pod "67469c9e-8e6c-4a36-a4e8-d85a1f010121" (UID: "67469c9e-8e6c-4a36-a4e8-d85a1f010121"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:03.590607 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.590561 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:03.603949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.603916 5978 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-cacerts\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:03.603949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.603945 5978 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-dns-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:03.603949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.603956 5978 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67469c9e-8e6c-4a36-a4e8-d85a1f010121-local-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:03.604203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.603968 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47wzm\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-kube-api-access-47wzm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:03.604203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.603977 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:03.604203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.603984 5978 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-kubeconfig\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:03.604203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.603992 5978 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67469c9e-8e6c-4a36-a4e8-d85a1f010121-istio-csr-ca-configmap\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:03.873704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.873669 5978 generic.go:356] "Generic (PLEG): container finished" podID="67469c9e-8e6c-4a36-a4e8-d85a1f010121" containerID="c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb" exitCode=0 Oct 31 01:23:03.873704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.873716 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" Oct 31 01:23:03.874009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.873734 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" event={"ID":"67469c9e-8e6c-4a36-a4e8-d85a1f010121","Type":"ContainerDied","Data":"c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb"} Oct 31 01:23:03.874009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.873785 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd" event={"ID":"67469c9e-8e6c-4a36-a4e8-d85a1f010121","Type":"ContainerDied","Data":"4117c23d4437e7c898a09af2115b1056ed29efa71946c4a4228a00a3dd9a6b2b"} Oct 31 01:23:03.874009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.873810 5978 scope.go:122] "RemoveContainer" containerID="c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb" Oct 31 01:23:03.885852 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.885830 5978 scope.go:122] "RemoveContainer" containerID="c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb" Oct 31 01:23:03.886151 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:03.886133 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb\": container with ID starting with c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb not found: ID does not exist" containerID="c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb" Oct 31 01:23:03.886193 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.886164 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb"} err="failed to get container status \"c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb\": rpc error: code = NotFound desc = could not find container \"c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb\": container with ID starting with c17ca73e58514acf59e9b4ae5e685674868092b9bbea6b8144343e6d21ee31cb not found: ID does not exist" Oct 31 01:23:03.921147 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.921114 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd"] Oct 31 01:23:03.930582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.930540 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sgntd"] Oct 31 01:23:03.947375 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:03.947326 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:04.115410 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:04.115373 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:04.283646 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:04.283557 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg"] Oct 31 01:23:04.284132 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:04.283828 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" containerID="cri-o://f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76" gracePeriod=30 Oct 31 01:23:04.370096 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:04.370044 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:04.436932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:04.436895 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67469c9e-8e6c-4a36-a4e8-d85a1f010121" path="/var/lib/kubelet/pods/67469c9e-8e6c-4a36-a4e8-d85a1f010121/volumes" Oct 31 01:23:04.797561 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:04.797530 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:05.585958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:05.585920 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:06.217694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:06.217651 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:06.371727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:06.371686 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:06.496825 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:06.496728 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5"] Oct 31 01:23:06.497026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:06.496975 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" containerID="cri-o://d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea" gracePeriod=30 Oct 31 01:23:06.569868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:06.569824 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:07.954686 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:07.954643 5978 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:23:07.955083 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:07.954705 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:23:07.970942 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:07.970906 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:09.534896 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.534842 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:09.585206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.585180 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.674957 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15d9e29-9873-4e96-bb76-d281d1a952ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15d9e29-9873-4e96-bb76-d281d1a952ff-istiod-ca-cert\") pod \"b15d9e29-9873-4e96-bb76-d281d1a952ff\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.675017 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-token\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-token\") pod \"b15d9e29-9873-4e96-bb76-d281d1a952ff\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.675083 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-socket\") pod \"b15d9e29-9873-4e96-bb76-d281d1a952ff\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.675241 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-certs\") pod \"b15d9e29-9873-4e96-bb76-d281d1a952ff\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.675301 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-podinfo\") pod \"b15d9e29-9873-4e96-bb76-d281d1a952ff\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.675343 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-credential-socket\") pod \"b15d9e29-9873-4e96-bb76-d281d1a952ff\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.675405 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-envoy\") pod \"b15d9e29-9873-4e96-bb76-d281d1a952ff\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.675513 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-kube-api-access-x2lpx\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-kube-api-access-x2lpx\") pod \"b15d9e29-9873-4e96-bb76-d281d1a952ff\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.675588 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-data\") pod \"b15d9e29-9873-4e96-bb76-d281d1a952ff\" (UID: \"b15d9e29-9873-4e96-bb76-d281d1a952ff\") " Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.675722 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15d9e29-9873-4e96-bb76-d281d1a952ff-istiod-ca-cert" pod "b15d9e29-9873-4e96-bb76-d281d1a952ff" (UID: "b15d9e29-9873-4e96-bb76-d281d1a952ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.676042 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-certs" pod "b15d9e29-9873-4e96-bb76-d281d1a952ff" (UID: "b15d9e29-9873-4e96-bb76-d281d1a952ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.676214 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.676233 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15d9e29-9873-4e96-bb76-d281d1a952ff-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:09.676548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.676441 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-credential-socket" pod "b15d9e29-9873-4e96-bb76-d281d1a952ff" (UID: "b15d9e29-9873-4e96-bb76-d281d1a952ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.682243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.676675 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-data" pod "b15d9e29-9873-4e96-bb76-d281d1a952ff" (UID: "b15d9e29-9873-4e96-bb76-d281d1a952ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.682243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.676746 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-socket" pod "b15d9e29-9873-4e96-bb76-d281d1a952ff" (UID: "b15d9e29-9873-4e96-bb76-d281d1a952ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.683893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.683858 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-podinfo" pod "b15d9e29-9873-4e96-bb76-d281d1a952ff" (UID: "b15d9e29-9873-4e96-bb76-d281d1a952ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:09.685982 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.684722 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-envoy" pod "b15d9e29-9873-4e96-bb76-d281d1a952ff" (UID: "b15d9e29-9873-4e96-bb76-d281d1a952ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:09.685982 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.684820 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-kube-api-access-x2lpx" pod "b15d9e29-9873-4e96-bb76-d281d1a952ff" (UID: "b15d9e29-9873-4e96-bb76-d281d1a952ff"). InnerVolumeSpecName "kube-api-access-x2lpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:09.686620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.686564 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-token" pod "b15d9e29-9873-4e96-bb76-d281d1a952ff" (UID: "b15d9e29-9873-4e96-bb76-d281d1a952ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:09.732300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.732254 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:09.743256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.743223 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc"] Oct 31 01:23:09.777420 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.777384 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:09.777420 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.777415 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2lpx\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-kube-api-access-x2lpx\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:09.777420 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.777425 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:09.777710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.777434 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:09.777710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.777442 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:09.777710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.777464 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15d9e29-9873-4e96-bb76-d281d1a952ff-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:09.777710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.777476 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15d9e29-9873-4e96-bb76-d281d1a952ff-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:09.879491 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.879431 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:09.912045 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.912006 5978 generic.go:356] "Generic (PLEG): container finished" podID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerID="f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76" exitCode=0 Oct 31 01:23:09.912206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.912064 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" Oct 31 01:23:09.912206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.912095 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" event={"ID":"b15d9e29-9873-4e96-bb76-d281d1a952ff","Type":"ContainerDied","Data":"f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76"} Oct 31 01:23:09.912206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.912160 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg" event={"ID":"b15d9e29-9873-4e96-bb76-d281d1a952ff","Type":"ContainerDied","Data":"7810b8cb199820d78f45664f1669afc8903adae560456adde1af97cf2d05e821"} Oct 31 01:23:09.912206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.912187 5978 scope.go:122] "RemoveContainer" containerID="f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76" Oct 31 01:23:09.922984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.922964 5978 scope.go:122] "RemoveContainer" containerID="f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76" Oct 31 01:23:09.923296 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:09.923274 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76\": container with ID starting with f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76 not found: ID does not exist" containerID="f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76" Oct 31 01:23:09.923347 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.923305 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76"} err="failed to get container status \"f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76\": rpc error: code = NotFound desc = could not find container \"f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76\": container with ID starting with f2669447a93456f2dbd006603961c62e4654be8912a53b5b0c13d466394ede76 not found: ID does not exist" Oct 31 01:23:09.960909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.960869 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg"] Oct 31 01:23:09.969167 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:09.969131 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ada0c1-openshift-default-79866ddb6c-knmhg"] Oct 31 01:23:10.019481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:10.019414 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:10.190245 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:10.190154 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:10.354808 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:10.354762 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:10.425989 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:10.425952 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" path="/var/lib/kubelet/pods/b15d9e29-9873-4e96-bb76-d281d1a952ff/volumes" Oct 31 01:23:10.504529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:10.504426 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:10.953300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:10.953266 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:11.786333 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.786263 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 01:23:11.902815 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.902772 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-data\") pod \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " Oct 31 01:23:11.903006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.902882 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-certs\") pod \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " Oct 31 01:23:11.903006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.902910 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-podinfo\") pod \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " Oct 31 01:23:11.903006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.902957 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-kube-api-access-7tzhn\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-kube-api-access-7tzhn\") pod \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " Oct 31 01:23:11.903116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903004 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-credential-socket\") pod \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " Oct 31 01:23:11.903116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903042 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istiod-ca-cert\") pod \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " Oct 31 01:23:11.903198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903114 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-socket\") pod \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " Oct 31 01:23:11.903198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903150 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-envoy\") pod \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " Oct 31 01:23:11.903198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903188 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-token\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-token\") pod \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\" (UID: \"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6\") " Oct 31 01:23:11.903944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903660 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-data" pod "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" (UID: "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:11.903944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903711 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-certs" pod "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" (UID: "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:11.903944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903724 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-credential-socket" pod "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" (UID: "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:11.903944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903913 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-socket" pod "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" (UID: "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:11.903944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.903941 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istiod-ca-cert" pod "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" (UID: "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:11.905955 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.905926 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-token" pod "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" (UID: "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:11.906115 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.906097 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-podinfo" pod "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" (UID: "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:11.906216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.906193 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-envoy" pod "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" (UID: "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:11.906216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.906206 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-kube-api-access-7tzhn" pod "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" (UID: "e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6"). InnerVolumeSpecName "kube-api-access-7tzhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:11.928415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.928384 5978 generic.go:356] "Generic (PLEG): container finished" podID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerID="d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea" exitCode=0 Oct 31 01:23:11.928601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.928436 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" Oct 31 01:23:11.928601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.928437 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" event={"ID":"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6","Type":"ContainerDied","Data":"d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea"} Oct 31 01:23:11.928601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.928574 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5" event={"ID":"e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6","Type":"ContainerDied","Data":"59c3b48ff29cc1db712faf1475ece7f83747f0853ff937ce822dab6d673177ee"} Oct 31 01:23:11.928601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.928599 5978 scope.go:122] "RemoveContainer" containerID="d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea" Oct 31 01:23:11.939704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.939682 5978 scope.go:122] "RemoveContainer" containerID="df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb" Oct 31 01:23:11.950409 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.950387 5978 scope.go:122] "RemoveContainer" containerID="d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea" Oct 31 01:23:11.950733 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:11.950710 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea\": container with ID starting with d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea not found: ID does not exist" containerID="d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea" Oct 31 01:23:11.950827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.950747 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea"} err="failed to get container status \"d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea\": rpc error: code = NotFound desc = could not find container \"d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea\": container with ID starting with d496aed4ecd033d719e4335b84a7e96d9d2ac696f6e33fd2d74025b5f3140bea not found: ID does not exist" Oct 31 01:23:11.950827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.950773 5978 scope.go:122] "RemoveContainer" containerID="df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb" Oct 31 01:23:11.951069 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:11.951027 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb\": container with ID starting with df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb not found: ID does not exist" containerID="df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb" Oct 31 01:23:11.951114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.951075 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb"} err="failed to get container status \"df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb\": rpc error: code = NotFound desc = could not find container \"df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb\": container with ID starting with df5321d0b0734f26c1f1c64d8da87344bfa10c40053d730154f835777422edbb not found: ID does not exist" Oct 31 01:23:11.974378 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.974347 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5"] Oct 31 01:23:11.980800 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:11.980769 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be301b-openshift-default-7bd4946d5f-z6hx5"] Oct 31 01:23:12.005070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.005026 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:12.005070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.005060 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:12.005070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.005074 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:12.005334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.005087 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tzhn\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-kube-api-access-7tzhn\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:12.005334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.005101 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:12.005334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.005112 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:12.005334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.005124 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:12.005334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.005135 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:12.005334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.005150 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:12.378551 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.378511 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:12.425678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:12.425639 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" path="/var/lib/kubelet/pods/e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6/volumes" Oct 31 01:23:14.200028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:14.199979 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:15.284176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:15.284138 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm"] Oct 31 01:23:15.313824 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:15.313707 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f"] Oct 31 01:23:15.456960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:15.456915 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:15.602366 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:15.602326 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:15.882835 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:15.882741 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:17.193051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.193017 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b"] Oct 31 01:23:17.193520 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.193233 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" podUID="da9da597-737d-470f-bfd0-507c7940f1f4" containerName="maas-api" containerID="cri-o://d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3" gracePeriod=30 Oct 31 01:23:17.305927 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.305889 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:17.478973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.478943 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:23:17.573493 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.573441 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9da597-737d-470f-bfd0-507c7940f1f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9da597-737d-470f-bfd0-507c7940f1f4-openshift-service-ca\") pod \"da9da597-737d-470f-bfd0-507c7940f1f4\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " Oct 31 01:23:17.573695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.573528 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9da597-737d-470f-bfd0-507c7940f1f4-kube-api-access-4rgrr\" (UniqueName: \"kubernetes.io/projected/da9da597-737d-470f-bfd0-507c7940f1f4-kube-api-access-4rgrr\") pod \"da9da597-737d-470f-bfd0-507c7940f1f4\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " Oct 31 01:23:17.573695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.573567 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-metrics-certs\") pod \"da9da597-737d-470f-bfd0-507c7940f1f4\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " Oct 31 01:23:17.573695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.573622 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-maas-api-tls\") pod \"da9da597-737d-470f-bfd0-507c7940f1f4\" (UID: \"da9da597-737d-470f-bfd0-507c7940f1f4\") " Oct 31 01:23:17.573863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.573807 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9da597-737d-470f-bfd0-507c7940f1f4-openshift-service-ca" pod "da9da597-737d-470f-bfd0-507c7940f1f4" (UID: "da9da597-737d-470f-bfd0-507c7940f1f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:17.574170 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.574122 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9da597-737d-470f-bfd0-507c7940f1f4-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:17.575932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.575901 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9da597-737d-470f-bfd0-507c7940f1f4-kube-api-access-4rgrr" pod "da9da597-737d-470f-bfd0-507c7940f1f4" (UID: "da9da597-737d-470f-bfd0-507c7940f1f4"). InnerVolumeSpecName "kube-api-access-4rgrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:17.576046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.575948 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-metrics-certs" pod "da9da597-737d-470f-bfd0-507c7940f1f4" (UID: "da9da597-737d-470f-bfd0-507c7940f1f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:17.576046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.575960 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-maas-api-tls" pod "da9da597-737d-470f-bfd0-507c7940f1f4" (UID: "da9da597-737d-470f-bfd0-507c7940f1f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:17.652609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.652564 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:17.674857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.674815 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rgrr\" (UniqueName: \"kubernetes.io/projected/da9da597-737d-470f-bfd0-507c7940f1f4-kube-api-access-4rgrr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:17.674857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.674848 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:17.674857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.674859 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9da597-737d-470f-bfd0-507c7940f1f4-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:17.864435 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.864403 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:17.962099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.962063 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp"] Oct 31 01:23:17.962352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.962327 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" containerID="cri-o://6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc" gracePeriod=30 Oct 31 01:23:17.975799 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.975766 5978 generic.go:356] "Generic (PLEG): container finished" podID="da9da597-737d-470f-bfd0-507c7940f1f4" containerID="d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3" exitCode=0 Oct 31 01:23:17.975956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.975817 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" Oct 31 01:23:17.975956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.975850 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" event={"ID":"da9da597-737d-470f-bfd0-507c7940f1f4","Type":"ContainerDied","Data":"d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3"} Oct 31 01:23:17.975956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.975907 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b" event={"ID":"da9da597-737d-470f-bfd0-507c7940f1f4","Type":"ContainerDied","Data":"6930aa5a03c8497d4a0cecf302b189d710e892f035da7562cf64b184cb6702fa"} Oct 31 01:23:17.975956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.975929 5978 scope.go:122] "RemoveContainer" containerID="d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3" Oct 31 01:23:17.987883 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.987862 5978 scope.go:122] "RemoveContainer" containerID="d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3" Oct 31 01:23:17.988211 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:17.988189 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3\": container with ID starting with d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3 not found: ID does not exist" containerID="d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3" Oct 31 01:23:17.988275 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:17.988221 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3"} err="failed to get container status \"d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3\": rpc error: code = NotFound desc = could not find container \"d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3\": container with ID starting with d3962d2a2942359739ffe6b17af9465f888aefbce917241ce5d66b01a179e9e3 not found: ID does not exist" Oct 31 01:23:18.020607 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:18.020566 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:18.041022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:18.040991 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b"] Oct 31 01:23:18.044571 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:18.044537 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-99a5c2-5ddbf99b76-tdq2b"] Oct 31 01:23:18.194153 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:18.194063 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:18.434703 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:18.434667 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9da597-737d-470f-bfd0-507c7940f1f4" path="/var/lib/kubelet/pods/da9da597-737d-470f-bfd0-507c7940f1f4/volumes" Oct 31 01:23:18.547740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:18.547640 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:19.883519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:19.883482 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:20.284518 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:20.284402 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" podUID="758d1b69-ab65-4337-9e7d-d24e7b5feda6" containerName="payload-processing" containerID="cri-o://8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316" gracePeriod=25 Oct 31 01:23:20.314093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:20.314042 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" podUID="47fce07c-1185-466e-be3b-7598826ddccc" containerName="payload-pre-processing" containerID="cri-o://8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710" gracePeriod=25 Oct 31 01:23:21.718140 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:21.718095 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:23.218124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.218097 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:23:23.336840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.336805 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-token\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-token\") pod \"03bfd765-42c6-483e-ae9b-df8aff4cff55\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " Oct 31 01:23:23.337063 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.336912 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-socket\") pod \"03bfd765-42c6-483e-ae9b-df8aff4cff55\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " Oct 31 01:23:23.337063 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337053 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-envoy\") pod \"03bfd765-42c6-483e-ae9b-df8aff4cff55\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " Oct 31 01:23:23.337194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337081 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-certs\") pod \"03bfd765-42c6-483e-ae9b-df8aff4cff55\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " Oct 31 01:23:23.337194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337120 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03bfd765-42c6-483e-ae9b-df8aff4cff55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bfd765-42c6-483e-ae9b-df8aff4cff55-istiod-ca-cert\") pod \"03bfd765-42c6-483e-ae9b-df8aff4cff55\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " Oct 31 01:23:23.337194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337155 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-credential-socket\") pod \"03bfd765-42c6-483e-ae9b-df8aff4cff55\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " Oct 31 01:23:23.337364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337204 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-data\") pod \"03bfd765-42c6-483e-ae9b-df8aff4cff55\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " Oct 31 01:23:23.337364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337247 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-socket" pod "03bfd765-42c6-483e-ae9b-df8aff4cff55" (UID: "03bfd765-42c6-483e-ae9b-df8aff4cff55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:23.337364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337260 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-kube-api-access-52nlw\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-kube-api-access-52nlw\") pod \"03bfd765-42c6-483e-ae9b-df8aff4cff55\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " Oct 31 01:23:23.337364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337310 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-podinfo\") pod \"03bfd765-42c6-483e-ae9b-df8aff4cff55\" (UID: \"03bfd765-42c6-483e-ae9b-df8aff4cff55\") " Oct 31 01:23:23.337631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337379 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-certs" pod "03bfd765-42c6-483e-ae9b-df8aff4cff55" (UID: "03bfd765-42c6-483e-ae9b-df8aff4cff55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:23.337631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337516 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-data" pod "03bfd765-42c6-483e-ae9b-df8aff4cff55" (UID: "03bfd765-42c6-483e-ae9b-df8aff4cff55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:23.337631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337568 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bfd765-42c6-483e-ae9b-df8aff4cff55-istiod-ca-cert" pod "03bfd765-42c6-483e-ae9b-df8aff4cff55" (UID: "03bfd765-42c6-483e-ae9b-df8aff4cff55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:23.337856 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337827 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-credential-socket" pod "03bfd765-42c6-483e-ae9b-df8aff4cff55" (UID: "03bfd765-42c6-483e-ae9b-df8aff4cff55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:23.337993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337918 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.337993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337935 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.337993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337943 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bfd765-42c6-483e-ae9b-df8aff4cff55-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.337993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.337953 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.339329 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.339300 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-envoy" pod "03bfd765-42c6-483e-ae9b-df8aff4cff55" (UID: "03bfd765-42c6-483e-ae9b-df8aff4cff55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:23.339481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.339331 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-token" pod "03bfd765-42c6-483e-ae9b-df8aff4cff55" (UID: "03bfd765-42c6-483e-ae9b-df8aff4cff55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:23.339481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.339369 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-kube-api-access-52nlw" pod "03bfd765-42c6-483e-ae9b-df8aff4cff55" (UID: "03bfd765-42c6-483e-ae9b-df8aff4cff55"). InnerVolumeSpecName "kube-api-access-52nlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:23.339985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.339969 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-podinfo" pod "03bfd765-42c6-483e-ae9b-df8aff4cff55" (UID: "03bfd765-42c6-483e-ae9b-df8aff4cff55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:23.439105 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.439072 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.439105 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.439101 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bfd765-42c6-483e-ae9b-df8aff4cff55-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.439105 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.439110 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52nlw\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-kube-api-access-52nlw\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.439369 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.439119 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.439369 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.439127 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bfd765-42c6-483e-ae9b-df8aff4cff55-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:23.800919 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:23.800834 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:24.015060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.015025 5978 generic.go:356] "Generic (PLEG): container finished" podID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerID="6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc" exitCode=0 Oct 31 01:23:24.015060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.015065 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" event={"ID":"03bfd765-42c6-483e-ae9b-df8aff4cff55","Type":"ContainerDied","Data":"6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc"} Oct 31 01:23:24.015304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.015080 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" Oct 31 01:23:24.015304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.015108 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp" event={"ID":"03bfd765-42c6-483e-ae9b-df8aff4cff55","Type":"ContainerDied","Data":"994e294ce433a404d792fde209248d468bccdc4f985f73518097bfd378ec16c3"} Oct 31 01:23:24.015304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.015131 5978 scope.go:122] "RemoveContainer" containerID="6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc" Oct 31 01:23:24.026439 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.026419 5978 scope.go:122] "RemoveContainer" containerID="dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede" Oct 31 01:23:24.037026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.036999 5978 scope.go:122] "RemoveContainer" containerID="6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc" Oct 31 01:23:24.037360 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:24.037342 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc\": container with ID starting with 6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc not found: ID does not exist" containerID="6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc" Oct 31 01:23:24.037433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.037369 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc"} err="failed to get container status \"6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc\": rpc error: code = NotFound desc = could not find container \"6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc\": container with ID starting with 6ceb2a06846cebfec5f16feb1171b3e3959db489d808d3d9c0c25e32f25816bc not found: ID does not exist" Oct 31 01:23:24.037433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.037389 5978 scope.go:122] "RemoveContainer" containerID="dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede" Oct 31 01:23:24.037672 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:24.037657 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede\": container with ID starting with dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede not found: ID does not exist" containerID="dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede" Oct 31 01:23:24.037716 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.037676 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede"} err="failed to get container status \"dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede\": rpc error: code = NotFound desc = could not find container \"dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede\": container with ID starting with dcdecb2b8157f3053006de924540bc8c6c28dad8ea0a91658cca4624ee268ede not found: ID does not exist" Oct 31 01:23:24.072588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.072557 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp"] Oct 31 01:23:24.075038 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.075007 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-114c95-openshift-default-68d89757b9-nxktp"] Oct 31 01:23:24.425135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.425056 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" path="/var/lib/kubelet/pods/03bfd765-42c6-483e-ae9b-df8aff4cff55/volumes" Oct 31 01:23:24.698282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.698204 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:24.766031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.765992 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" containerID="cri-o://73703e0dc337d928d4915e9da5a9e293d9e441edbc525f9d3f1ea7190c4729d4" gracePeriod=45 Oct 31 01:23:24.846124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:24.846086 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:25.025090 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.025055 5978 generic.go:356] "Generic (PLEG): container finished" podID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerID="73703e0dc337d928d4915e9da5a9e293d9e441edbc525f9d3f1ea7190c4729d4" exitCode=0 Oct 31 01:23:25.025235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.025119 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" event={"ID":"02f2f37e-6172-4c79-8973-cfdf4c660bfd","Type":"ContainerDied","Data":"73703e0dc337d928d4915e9da5a9e293d9e441edbc525f9d3f1ea7190c4729d4"} Oct 31 01:23:25.042683 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.042658 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 01:23:25.075157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.075106 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:25.159005 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.158968 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tmp-dir\") pod \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " Oct 31 01:23:25.159005 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159010 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f2f37e-6172-4c79-8973-cfdf4c660bfd-kube-api-access-s76g6\" (UniqueName: \"kubernetes.io/projected/02f2f37e-6172-4c79-8973-cfdf4c660bfd-kube-api-access-s76g6\") pod \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " Oct 31 01:23:25.159238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159086 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-home\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-home\") pod \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " Oct 31 01:23:25.159238 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159163 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tls-certs\" (UniqueName: \"kubernetes.io/secret/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tls-certs\") pod \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " Oct 31 01:23:25.159357 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159239 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-dshm\") pod \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " Oct 31 01:23:25.159357 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159348 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-model-cache\") pod \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\" (UID: \"02f2f37e-6172-4c79-8973-cfdf4c660bfd\") " Oct 31 01:23:25.159576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159445 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-home" pod "02f2f37e-6172-4c79-8973-cfdf4c660bfd" (UID: "02f2f37e-6172-4c79-8973-cfdf4c660bfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.159576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159342 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tmp-dir" pod "02f2f37e-6172-4c79-8973-cfdf4c660bfd" (UID: "02f2f37e-6172-4c79-8973-cfdf4c660bfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.159789 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159763 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-model-cache" pod "02f2f37e-6172-4c79-8973-cfdf4c660bfd" (UID: "02f2f37e-6172-4c79-8973-cfdf4c660bfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.159865 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159843 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.159925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159865 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.159925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.159878 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.161546 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.161513 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-dshm" pod "02f2f37e-6172-4c79-8973-cfdf4c660bfd" (UID: "02f2f37e-6172-4c79-8973-cfdf4c660bfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:25.161655 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.161556 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tls-certs" pod "02f2f37e-6172-4c79-8973-cfdf4c660bfd" (UID: "02f2f37e-6172-4c79-8973-cfdf4c660bfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:25.161655 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.161634 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f2f37e-6172-4c79-8973-cfdf4c660bfd-kube-api-access-s76g6" pod "02f2f37e-6172-4c79-8973-cfdf4c660bfd" (UID: "02f2f37e-6172-4c79-8973-cfdf4c660bfd"). InnerVolumeSpecName "kube-api-access-s76g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:25.260970 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.260930 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02f2f37e-6172-4c79-8973-cfdf4c660bfd-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.260970 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.260968 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s76g6\" (UniqueName: \"kubernetes.io/projected/02f2f37e-6172-4c79-8973-cfdf4c660bfd-kube-api-access-s76g6\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.261220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.260982 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02f2f37e-6172-4c79-8973-cfdf4c660bfd-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:25.585037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:25.584980 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:26.035060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:26.034970 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" Oct 31 01:23:26.035346 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:26.035007 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc" event={"ID":"02f2f37e-6172-4c79-8973-cfdf4c660bfd","Type":"ContainerDied","Data":"c378d1eeb706cf165533bd028c0b41e99c696749ad5e9768bd9b977346b13e6f"} Oct 31 01:23:26.035515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:26.035382 5978 scope.go:122] "RemoveContainer" containerID="73703e0dc337d928d4915e9da5a9e293d9e441edbc525f9d3f1ea7190c4729d4" Oct 31 01:23:26.093385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:26.093350 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc"] Oct 31 01:23:26.102432 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:26.102398 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-daaf26/premium-w3-daaf26-kserve-7f4475df7f-wmxlc"] Oct 31 01:23:26.425058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:26.425024 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" path="/var/lib/kubelet/pods/02f2f37e-6172-4c79-8973-cfdf4c660bfd/volumes" Oct 31 01:23:26.711849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:26.711755 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" podUID="47fce07c-1185-466e-be3b-7598826ddccc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:23:26.711849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:26.711792 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" podUID="758d1b69-ab65-4337-9e7d-d24e7b5feda6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:23:27.128979 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.128936 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65"] Oct 31 01:23:27.129563 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129544 5978 cpu_manager.go:414] "containerMap: removing container" podUID="da9da597-737d-470f-bfd0-507c7940f1f4" containerName="maas-api" Oct 31 01:23:27.129563 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129562 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9da597-737d-470f-bfd0-507c7940f1f4" containerName="maas-api" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129576 5978 cpu_manager.go:414] "containerMap: removing container" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129581 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129598 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129603 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129613 5978 cpu_manager.go:414] "containerMap: removing container" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129621 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129656 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129663 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129677 5978 cpu_manager.go:414] "containerMap: removing container" podUID="41d84e9d-39fe-400a-9981-f4588885fa53" containerName="maas-api" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129684 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d84e9d-39fe-400a-9981-f4588885fa53" containerName="maas-api" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129700 5978 cpu_manager.go:414] "containerMap: removing container" podUID="67469c9e-8e6c-4a36-a4e8-d85a1f010121" containerName="discovery" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129706 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67469c9e-8e6c-4a36-a4e8-d85a1f010121" containerName="discovery" Oct 31 01:23:27.129721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129723 5978 cpu_manager.go:414] "containerMap: removing container" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129730 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129876 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="67469c9e-8e6c-4a36-a4e8-d85a1f010121" containerName="discovery" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129895 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f2f37e-6172-4c79-8973-cfdf4c660bfd" containerName="main" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129908 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9da597-737d-470f-bfd0-507c7940f1f4" containerName="maas-api" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129916 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129927 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129933 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bfd765-42c6-483e-ae9b-df8aff4cff55" containerName="istio-proxy" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129947 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129955 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15d9e29-9873-4e96-bb76-d281d1a952ff" containerName="istio-proxy" Oct 31 01:23:27.130143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.129964 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d84e9d-39fe-400a-9981-f4588885fa53" containerName="maas-api" Oct 31 01:23:27.134776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.134752 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.147871 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.147838 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:23:27.148075 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.148058 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:23:27.160623 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.160584 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65"] Oct 31 01:23:27.183937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.183894 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8dpt\" (UniqueName: \"kubernetes.io/projected/0d5e32a8-8cec-42f8-8264-35956dca8e0c-kube-api-access-t8dpt\") pod \"maas-api-revoke-sub-6cc4b638-26s65\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.184149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.184034 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc4b638-26s65\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.184149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.184119 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc4b638-26s65\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.285427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.285389 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8dpt\" (UniqueName: \"kubernetes.io/projected/0d5e32a8-8cec-42f8-8264-35956dca8e0c-kube-api-access-t8dpt\") pod \"maas-api-revoke-sub-6cc4b638-26s65\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.285676 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.285499 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc4b638-26s65\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.285676 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.285564 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc4b638-26s65\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.286122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.286101 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc4b638-26s65\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.287987 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.287962 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc4b638-26s65\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.311408 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.311370 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8dpt\" (UniqueName: \"kubernetes.io/projected/0d5e32a8-8cec-42f8-8264-35956dca8e0c-kube-api-access-t8dpt\") pod \"maas-api-revoke-sub-6cc4b638-26s65\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.448890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.448799 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:27.534440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.534363 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:27.751719 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:27.751692 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65"] Oct 31 01:23:27.754623 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:23:27.754591 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5e32a8_8cec_42f8_8264_35956dca8e0c.slice/crio-5e9c32932693c13b9ad2506f7502cd0220a0c504e34cece6c3ecd4372b01468f WatchSource:0}: Error finding container 5e9c32932693c13b9ad2506f7502cd0220a0c504e34cece6c3ecd4372b01468f: Status 404 returned error can't find the container with id 5e9c32932693c13b9ad2506f7502cd0220a0c504e34cece6c3ecd4372b01468f Oct 31 01:23:28.050232 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:28.050191 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" event={"ID":"0d5e32a8-8cec-42f8-8264-35956dca8e0c","Type":"ContainerStarted","Data":"363c69ea0f656b9eda5f0d21ae579d407f992addb189f40d51e59b23432d898e"} Oct 31 01:23:28.050438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:28.050256 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" event={"ID":"0d5e32a8-8cec-42f8-8264-35956dca8e0c","Type":"ContainerStarted","Data":"5e9c32932693c13b9ad2506f7502cd0220a0c504e34cece6c3ecd4372b01468f"} Oct 31 01:23:28.088811 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:28.088760 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" podStartSLOduration=1.08874566 podStartE2EDuration="1.08874566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:28.087018463 +0000 UTC m=+3825.785132227" watchObservedRunningTime="2026-10-31 01:23:28.08874566 +0000 UTC m=+3825.786859425" Oct 31 01:23:29.058495 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:29.058394 5978 generic.go:356] "Generic (PLEG): container finished" podID="0d5e32a8-8cec-42f8-8264-35956dca8e0c" containerID="363c69ea0f656b9eda5f0d21ae579d407f992addb189f40d51e59b23432d898e" exitCode=0 Oct 31 01:23:29.058900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:29.058487 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" event={"ID":"0d5e32a8-8cec-42f8-8264-35956dca8e0c","Type":"ContainerDied","Data":"363c69ea0f656b9eda5f0d21ae579d407f992addb189f40d51e59b23432d898e"} Oct 31 01:23:29.393086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:29.393048 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:30.202586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.202559 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:30.318808 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.318777 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-openshift-ca\") pod \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " Oct 31 01:23:30.318998 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.318822 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5e32a8-8cec-42f8-8264-35956dca8e0c-kube-api-access-t8dpt\" (UniqueName: \"kubernetes.io/projected/0d5e32a8-8cec-42f8-8264-35956dca8e0c-kube-api-access-t8dpt\") pod \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " Oct 31 01:23:30.318998 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.318891 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-xks-ca\") pod \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\" (UID: \"0d5e32a8-8cec-42f8-8264-35956dca8e0c\") " Oct 31 01:23:30.319283 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.319239 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-openshift-ca" pod "0d5e32a8-8cec-42f8-8264-35956dca8e0c" (UID: "0d5e32a8-8cec-42f8-8264-35956dca8e0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:30.321182 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.321151 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-xks-ca" pod "0d5e32a8-8cec-42f8-8264-35956dca8e0c" (UID: "0d5e32a8-8cec-42f8-8264-35956dca8e0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:30.321182 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.321160 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5e32a8-8cec-42f8-8264-35956dca8e0c-kube-api-access-t8dpt" pod "0d5e32a8-8cec-42f8-8264-35956dca8e0c" (UID: "0d5e32a8-8cec-42f8-8264-35956dca8e0c"). InnerVolumeSpecName "kube-api-access-t8dpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:30.420602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.420571 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.420602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.420599 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8dpt\" (UniqueName: \"kubernetes.io/projected/0d5e32a8-8cec-42f8-8264-35956dca8e0c-kube-api-access-t8dpt\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.420602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.420609 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5e32a8-8cec-42f8-8264-35956dca8e0c-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:30.997735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:30.997677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:31.073604 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:31.073565 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" Oct 31 01:23:31.073604 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:31.073585 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65" event={"ID":"0d5e32a8-8cec-42f8-8264-35956dca8e0c","Type":"ContainerDied","Data":"5e9c32932693c13b9ad2506f7502cd0220a0c504e34cece6c3ecd4372b01468f"} Oct 31 01:23:31.073838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:31.073640 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e9c32932693c13b9ad2506f7502cd0220a0c504e34cece6c3ecd4372b01468f" Oct 31 01:23:32.693177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:32.693124 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:33.180681 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:33.180646 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:23:34.161631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.161590 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:34.211396 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.211352 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn"] Oct 31 01:23:34.211608 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.211591 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" podUID="5e5f0fde-ba78-4c22-82fd-35547be085d1" containerName="payload-processing" containerID="cri-o://e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8" gracePeriod=30 Oct 31 01:23:34.229682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.229652 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms"] Oct 31 01:23:34.229891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.229868 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" podUID="ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea" containerName="payload-pre-processing" containerID="cri-o://c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314" gracePeriod=30 Oct 31 01:23:34.328446 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.328409 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:34.489240 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.489205 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:34.514624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.514598 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:23:34.554415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.554334 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:23:34.668662 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.668631 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5f0fde-ba78-4c22-82fd-35547be085d1-kube-api-access-xk9kp\" (UniqueName: \"kubernetes.io/projected/5e5f0fde-ba78-4c22-82fd-35547be085d1-kube-api-access-xk9kp\") pod \"5e5f0fde-ba78-4c22-82fd-35547be085d1\" (UID: \"5e5f0fde-ba78-4c22-82fd-35547be085d1\") " Oct 31 01:23:34.668857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.668682 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5f0fde-ba78-4c22-82fd-35547be085d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5f0fde-ba78-4c22-82fd-35547be085d1-plugins-config-volume\") pod \"5e5f0fde-ba78-4c22-82fd-35547be085d1\" (UID: \"5e5f0fde-ba78-4c22-82fd-35547be085d1\") " Oct 31 01:23:34.668857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.668718 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-plugins-config-volume\") pod \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\" (UID: \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\") " Oct 31 01:23:34.668857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.668759 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-kube-api-access-6x2hj\" (UniqueName: \"kubernetes.io/projected/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-kube-api-access-6x2hj\") pod \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\" (UID: \"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea\") " Oct 31 01:23:34.669173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.669147 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5f0fde-ba78-4c22-82fd-35547be085d1-plugins-config-volume" pod "5e5f0fde-ba78-4c22-82fd-35547be085d1" (UID: "5e5f0fde-ba78-4c22-82fd-35547be085d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:34.669173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.669153 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-plugins-config-volume" pod "ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea" (UID: "ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:34.669358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.669340 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e5f0fde-ba78-4c22-82fd-35547be085d1-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:34.669433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.669363 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:34.671395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.671375 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5f0fde-ba78-4c22-82fd-35547be085d1-kube-api-access-xk9kp" pod "5e5f0fde-ba78-4c22-82fd-35547be085d1" (UID: "5e5f0fde-ba78-4c22-82fd-35547be085d1"). InnerVolumeSpecName "kube-api-access-xk9kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:34.671533 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.671402 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-kube-api-access-6x2hj" pod "ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea" (UID: "ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea"). InnerVolumeSpecName "kube-api-access-6x2hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:34.739602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.739512 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:34.770894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.770861 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk9kp\" (UniqueName: \"kubernetes.io/projected/5e5f0fde-ba78-4c22-82fd-35547be085d1-kube-api-access-xk9kp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:34.770894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.770890 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x2hj\" (UniqueName: \"kubernetes.io/projected/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea-kube-api-access-6x2hj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:34.968862 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:34.968824 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:35.101784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.101753 5978 generic.go:356] "Generic (PLEG): container finished" podID="ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea" containerID="c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314" exitCode=0 Oct 31 01:23:35.101993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.101820 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" Oct 31 01:23:35.101993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.101835 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" event={"ID":"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea","Type":"ContainerDied","Data":"c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314"} Oct 31 01:23:35.101993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.101921 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms" event={"ID":"ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea","Type":"ContainerDied","Data":"f7b0de129b7d838a482029105c58c8adc320a41d3fe8703393fbae97b7bc875b"} Oct 31 01:23:35.101993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.101945 5978 scope.go:122] "RemoveContainer" containerID="c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314" Oct 31 01:23:35.103339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.103307 5978 generic.go:356] "Generic (PLEG): container finished" podID="5e5f0fde-ba78-4c22-82fd-35547be085d1" containerID="e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8" exitCode=0 Oct 31 01:23:35.103433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.103359 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" Oct 31 01:23:35.103433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.103385 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" event={"ID":"5e5f0fde-ba78-4c22-82fd-35547be085d1","Type":"ContainerDied","Data":"e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8"} Oct 31 01:23:35.103433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.103428 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn" event={"ID":"5e5f0fde-ba78-4c22-82fd-35547be085d1","Type":"ContainerDied","Data":"e7a44d44488e4d6af44422055aab18338ed994b166bdbbc3eb47e11e8b50e5b9"} Oct 31 01:23:35.113022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.113002 5978 scope.go:122] "RemoveContainer" containerID="c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314" Oct 31 01:23:35.113364 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:35.113333 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314\": container with ID starting with c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314 not found: ID does not exist" containerID="c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314" Oct 31 01:23:35.113478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.113374 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314"} err="failed to get container status \"c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314\": rpc error: code = NotFound desc = could not find container \"c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314\": container with ID starting with c5392d26b36fe110b1a492b0183e52b32316f6327775f6ff9e610052de261314 not found: ID does not exist" Oct 31 01:23:35.113478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.113397 5978 scope.go:122] "RemoveContainer" containerID="e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8" Oct 31 01:23:35.122464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.122434 5978 scope.go:122] "RemoveContainer" containerID="e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8" Oct 31 01:23:35.122726 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:35.122709 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8\": container with ID starting with e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8 not found: ID does not exist" containerID="e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8" Oct 31 01:23:35.122778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.122732 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8"} err="failed to get container status \"e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8\": rpc error: code = NotFound desc = could not find container \"e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8\": container with ID starting with e295478e5e5d1321662f09173b2cfc38d7bb87b5ab11d0cb14fe63b89a7371c8 not found: ID does not exist" Oct 31 01:23:35.151320 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.151280 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms"] Oct 31 01:23:35.159629 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.159601 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f0029-7fdf69d77f-7tcms"] Oct 31 01:23:35.185759 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.185729 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn"] Oct 31 01:23:35.190514 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.190489 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f0029-5b558588db-d5gpn"] Oct 31 01:23:35.423047 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:35.422957 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:36.408126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:36.408090 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:36.425340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:36.425305 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5f0fde-ba78-4c22-82fd-35547be085d1" path="/var/lib/kubelet/pods/5e5f0fde-ba78-4c22-82fd-35547be085d1/volumes" Oct 31 01:23:36.425753 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:36.425739 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea" path="/var/lib/kubelet/pods/ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea/volumes" Oct 31 01:23:36.559555 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:36.559517 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:36.712239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:36.712125 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" podUID="47fce07c-1185-466e-be3b-7598826ddccc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:23:36.712239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:36.712153 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" podUID="758d1b69-ab65-4337-9e7d-d24e7b5feda6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:23:36.712923 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:36.712899 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:38.017715 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.017682 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n"] Oct 31 01:23:38.018165 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.017926 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" podUID="440f07dc-e69f-4e51-b1de-178d98232462" containerName="maas-api" containerID="cri-o://df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff" gracePeriod=30 Oct 31 01:23:38.115406 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.115365 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:38.293816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.293786 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 01:23:38.414020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.413988 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440f07dc-e69f-4e51-b1de-178d98232462-kube-api-access-zmzkj\" (UniqueName: \"kubernetes.io/projected/440f07dc-e69f-4e51-b1de-178d98232462-kube-api-access-zmzkj\") pod \"440f07dc-e69f-4e51-b1de-178d98232462\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " Oct 31 01:23:38.414227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.414057 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440f07dc-e69f-4e51-b1de-178d98232462-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f07dc-e69f-4e51-b1de-178d98232462-openshift-service-ca\") pod \"440f07dc-e69f-4e51-b1de-178d98232462\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " Oct 31 01:23:38.414227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.414192 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-metrics-certs\") pod \"440f07dc-e69f-4e51-b1de-178d98232462\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " Oct 31 01:23:38.414360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.414272 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls\") pod \"440f07dc-e69f-4e51-b1de-178d98232462\" (UID: \"440f07dc-e69f-4e51-b1de-178d98232462\") " Oct 31 01:23:38.414416 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.414385 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440f07dc-e69f-4e51-b1de-178d98232462-openshift-service-ca" pod "440f07dc-e69f-4e51-b1de-178d98232462" (UID: "440f07dc-e69f-4e51-b1de-178d98232462"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:38.414712 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.414694 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f07dc-e69f-4e51-b1de-178d98232462-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:38.416313 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.416286 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440f07dc-e69f-4e51-b1de-178d98232462-kube-api-access-zmzkj" pod "440f07dc-e69f-4e51-b1de-178d98232462" (UID: "440f07dc-e69f-4e51-b1de-178d98232462"). InnerVolumeSpecName "kube-api-access-zmzkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:38.416440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.416328 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls" pod "440f07dc-e69f-4e51-b1de-178d98232462" (UID: "440f07dc-e69f-4e51-b1de-178d98232462"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:38.416440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.416401 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-metrics-certs" pod "440f07dc-e69f-4e51-b1de-178d98232462" (UID: "440f07dc-e69f-4e51-b1de-178d98232462"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:38.516206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.516164 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:38.516206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.516201 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f07dc-e69f-4e51-b1de-178d98232462-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:38.516206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.516212 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmzkj\" (UniqueName: \"kubernetes.io/projected/440f07dc-e69f-4e51-b1de-178d98232462-kube-api-access-zmzkj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:38.613721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.613668 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:38.765937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:38.765893 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:39.135910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.135872 5978 generic.go:356] "Generic (PLEG): container finished" podID="440f07dc-e69f-4e51-b1de-178d98232462" containerID="df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff" exitCode=0 Oct 31 01:23:39.136397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.135924 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" Oct 31 01:23:39.136397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.135958 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" event={"ID":"440f07dc-e69f-4e51-b1de-178d98232462","Type":"ContainerDied","Data":"df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff"} Oct 31 01:23:39.136397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.136020 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n" event={"ID":"440f07dc-e69f-4e51-b1de-178d98232462","Type":"ContainerDied","Data":"4e12b0ee43dc6c49f1cf014bb128c3591e4e698004f2ff847860dbd1c0fac880"} Oct 31 01:23:39.136397 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.136049 5978 scope.go:122] "RemoveContainer" containerID="df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff" Oct 31 01:23:39.146188 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.146165 5978 scope.go:122] "RemoveContainer" containerID="df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff" Oct 31 01:23:39.146581 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:39.146556 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff\": container with ID starting with df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff not found: ID does not exist" containerID="df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff" Oct 31 01:23:39.146666 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.146589 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff"} err="failed to get container status \"df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff\": rpc error: code = NotFound desc = could not find container \"df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff\": container with ID starting with df791d9fdbddf5c069930b604710e0a8da2f9e42bddeea5107f05e2592a321ff not found: ID does not exist" Oct 31 01:23:39.183512 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.183474 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n"] Oct 31 01:23:39.196729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.196692 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-daaf26-dcc9f44b6-zjn8n"] Oct 31 01:23:39.278398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:39.278364 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:40.424642 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.424614 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440f07dc-e69f-4e51-b1de-178d98232462" path="/var/lib/kubelet/pods/440f07dc-e69f-4e51-b1de-178d98232462/volumes" Oct 31 01:23:40.517057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.516967 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:40.579861 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.579832 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 01:23:40.605946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.605920 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 01:23:40.742115 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.742084 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758d1b69-ab65-4337-9e7d-d24e7b5feda6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758d1b69-ab65-4337-9e7d-d24e7b5feda6-plugins-config-volume\") pod \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\" (UID: \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\") " Oct 31 01:23:40.742300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.742146 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47fce07c-1185-466e-be3b-7598826ddccc-kube-api-access-vx2mv\" (UniqueName: \"kubernetes.io/projected/47fce07c-1185-466e-be3b-7598826ddccc-kube-api-access-vx2mv\") pod \"47fce07c-1185-466e-be3b-7598826ddccc\" (UID: \"47fce07c-1185-466e-be3b-7598826ddccc\") " Oct 31 01:23:40.742300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.742279 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758d1b69-ab65-4337-9e7d-d24e7b5feda6-kube-api-access-2m8fk\" (UniqueName: \"kubernetes.io/projected/758d1b69-ab65-4337-9e7d-d24e7b5feda6-kube-api-access-2m8fk\") pod \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\" (UID: \"758d1b69-ab65-4337-9e7d-d24e7b5feda6\") " Oct 31 01:23:40.742415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.742398 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47fce07c-1185-466e-be3b-7598826ddccc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fce07c-1185-466e-be3b-7598826ddccc-plugins-config-volume\") pod \"47fce07c-1185-466e-be3b-7598826ddccc\" (UID: \"47fce07c-1185-466e-be3b-7598826ddccc\") " Oct 31 01:23:40.742554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.742523 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758d1b69-ab65-4337-9e7d-d24e7b5feda6-plugins-config-volume" pod "758d1b69-ab65-4337-9e7d-d24e7b5feda6" (UID: "758d1b69-ab65-4337-9e7d-d24e7b5feda6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:40.742813 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.742794 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758d1b69-ab65-4337-9e7d-d24e7b5feda6-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:40.742940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.742822 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fce07c-1185-466e-be3b-7598826ddccc-plugins-config-volume" pod "47fce07c-1185-466e-be3b-7598826ddccc" (UID: "47fce07c-1185-466e-be3b-7598826ddccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:40.744569 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.744540 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758d1b69-ab65-4337-9e7d-d24e7b5feda6-kube-api-access-2m8fk" pod "758d1b69-ab65-4337-9e7d-d24e7b5feda6" (UID: "758d1b69-ab65-4337-9e7d-d24e7b5feda6"). InnerVolumeSpecName "kube-api-access-2m8fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:40.744665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.744543 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fce07c-1185-466e-be3b-7598826ddccc-kube-api-access-vx2mv" pod "47fce07c-1185-466e-be3b-7598826ddccc" (UID: "47fce07c-1185-466e-be3b-7598826ddccc"). InnerVolumeSpecName "kube-api-access-vx2mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:40.844254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.844211 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47fce07c-1185-466e-be3b-7598826ddccc-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:40.844254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.844245 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2mv\" (UniqueName: \"kubernetes.io/projected/47fce07c-1185-466e-be3b-7598826ddccc-kube-api-access-vx2mv\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:40.844254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:40.844256 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m8fk\" (UniqueName: \"kubernetes.io/projected/758d1b69-ab65-4337-9e7d-d24e7b5feda6-kube-api-access-2m8fk\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:41.157350 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.157261 5978 generic.go:356] "Generic (PLEG): container finished" podID="47fce07c-1185-466e-be3b-7598826ddccc" containerID="8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710" exitCode=0 Oct 31 01:23:41.157350 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.157325 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" Oct 31 01:23:41.157643 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.157348 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" event={"ID":"47fce07c-1185-466e-be3b-7598826ddccc","Type":"ContainerDied","Data":"8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710"} Oct 31 01:23:41.157643 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.157431 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f" event={"ID":"47fce07c-1185-466e-be3b-7598826ddccc","Type":"ContainerDied","Data":"a80164dd239928c4e1b380c4dd1b181cff107b3dd8c0be3e92bd398ef0c4f50f"} Oct 31 01:23:41.157643 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.157478 5978 scope.go:122] "RemoveContainer" containerID="8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710" Oct 31 01:23:41.159007 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.158979 5978 generic.go:356] "Generic (PLEG): container finished" podID="758d1b69-ab65-4337-9e7d-d24e7b5feda6" containerID="8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316" exitCode=0 Oct 31 01:23:41.159149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.159010 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" event={"ID":"758d1b69-ab65-4337-9e7d-d24e7b5feda6","Type":"ContainerDied","Data":"8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316"} Oct 31 01:23:41.159149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.159047 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" Oct 31 01:23:41.159149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.159060 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm" event={"ID":"758d1b69-ab65-4337-9e7d-d24e7b5feda6","Type":"ContainerDied","Data":"457ecf4528af6d6012888b674e3881792d59d9aa8b1940a9173c52a3a59f9e31"} Oct 31 01:23:41.171526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.171501 5978 scope.go:122] "RemoveContainer" containerID="8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710" Oct 31 01:23:41.171891 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:41.171865 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710\": container with ID starting with 8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710 not found: ID does not exist" containerID="8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710" Oct 31 01:23:41.171966 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.171901 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710"} err="failed to get container status \"8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710\": rpc error: code = NotFound desc = could not find container \"8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710\": container with ID starting with 8cd9f975125dc1ec5796ac6525e52443f473c27924036e27b76bf1c4674f3710 not found: ID does not exist" Oct 31 01:23:41.171966 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.171929 5978 scope.go:122] "RemoveContainer" containerID="8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316" Oct 31 01:23:41.182004 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.181983 5978 scope.go:122] "RemoveContainer" containerID="8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316" Oct 31 01:23:41.182297 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:41.182276 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316\": container with ID starting with 8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316 not found: ID does not exist" containerID="8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316" Oct 31 01:23:41.182344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.182308 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316"} err="failed to get container status \"8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316\": rpc error: code = NotFound desc = could not find container \"8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316\": container with ID starting with 8dbdd464f3894a54b32f59a0945557ab2ca3a427c9fdab0452519e968ad9d316 not found: ID does not exist" Oct 31 01:23:41.262098 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.262062 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f"] Oct 31 01:23:41.278234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.278197 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-daaf26-565c84fdcf-fq96f"] Oct 31 01:23:41.306096 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.306063 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm"] Oct 31 01:23:41.311609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:41.311581 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-daaf26-86c9494789-zkrtm"] Oct 31 01:23:42.292288 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.292255 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:42.425509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.425477 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47fce07c-1185-466e-be3b-7598826ddccc" path="/var/lib/kubelet/pods/47fce07c-1185-466e-be3b-7598826ddccc/volumes" Oct 31 01:23:42.425894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.425880 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758d1b69-ab65-4337-9e7d-d24e7b5feda6" path="/var/lib/kubelet/pods/758d1b69-ab65-4337-9e7d-d24e7b5feda6/volumes" Oct 31 01:23:42.463887 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.463850 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:42.618707 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.618668 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:42.778290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.778246 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s"] Oct 31 01:23:42.778542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.778503 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerName="istio-proxy" containerID="cri-o://4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675" gracePeriod=30 Oct 31 01:23:42.787031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.786999 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc"] Oct 31 01:23:42.787675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.787248 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerName="istio-proxy" containerID="cri-o://fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee" gracePeriod=30 Oct 31 01:23:42.790311 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.790281 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb"] Oct 31 01:23:42.790535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.790509 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" containerID="cri-o://1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90" gracePeriod=30 Oct 31 01:23:42.880341 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:42.880240 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:43.349073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.349034 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s"] Oct 31 01:23:43.350092 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350071 5978 cpu_manager.go:414] "containerMap: removing container" podUID="47fce07c-1185-466e-be3b-7598826ddccc" containerName="payload-pre-processing" Oct 31 01:23:43.350092 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350092 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47fce07c-1185-466e-be3b-7598826ddccc" containerName="payload-pre-processing" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350106 5978 cpu_manager.go:414] "containerMap: removing container" podUID="0d5e32a8-8cec-42f8-8264-35956dca8e0c" containerName="revoke-keys" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350112 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5e32a8-8cec-42f8-8264-35956dca8e0c" containerName="revoke-keys" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350123 5978 cpu_manager.go:414] "containerMap: removing container" podUID="440f07dc-e69f-4e51-b1de-178d98232462" containerName="maas-api" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350128 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440f07dc-e69f-4e51-b1de-178d98232462" containerName="maas-api" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350153 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350157 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a07c21-7ad3-4153-a7fe-a5ea3e263cc6" containerName="istio-proxy" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350176 5978 cpu_manager.go:414] "containerMap: removing container" podUID="ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea" containerName="payload-pre-processing" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350182 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea" containerName="payload-pre-processing" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350197 5978 cpu_manager.go:414] "containerMap: removing container" podUID="758d1b69-ab65-4337-9e7d-d24e7b5feda6" containerName="payload-processing" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350202 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758d1b69-ab65-4337-9e7d-d24e7b5feda6" containerName="payload-processing" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350210 5978 cpu_manager.go:414] "containerMap: removing container" podUID="5e5f0fde-ba78-4c22-82fd-35547be085d1" containerName="payload-processing" Oct 31 01:23:43.350220 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350214 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5f0fde-ba78-4c22-82fd-35547be085d1" containerName="payload-processing" Oct 31 01:23:43.350635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350318 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5f0fde-ba78-4c22-82fd-35547be085d1" containerName="payload-processing" Oct 31 01:23:43.350635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350328 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="47fce07c-1185-466e-be3b-7598826ddccc" containerName="payload-pre-processing" Oct 31 01:23:43.350635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350336 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5e32a8-8cec-42f8-8264-35956dca8e0c" containerName="revoke-keys" Oct 31 01:23:43.350635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350344 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="440f07dc-e69f-4e51-b1de-178d98232462" containerName="maas-api" Oct 31 01:23:43.350635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350354 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab46bd84-b8e6-4b9e-a5eb-56ec944e26ea" containerName="payload-pre-processing" Oct 31 01:23:43.350635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.350362 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="758d1b69-ab65-4337-9e7d-d24e7b5feda6" containerName="payload-processing" Oct 31 01:23:43.355283 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.355261 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.374170 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.374136 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s"] Oct 31 01:23:43.380094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.380060 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.380264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.380136 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.380264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.380245 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5v2p\" (UniqueName: \"kubernetes.io/projected/91b6d2a6-3fce-4c5b-91e5-3353442a6243-kube-api-access-d5v2p\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.444480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.444435 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:23:43.444658 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.444528 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:23:43.480882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.480851 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5v2p\" (UniqueName: \"kubernetes.io/projected/91b6d2a6-3fce-4c5b-91e5-3353442a6243-kube-api-access-d5v2p\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.481099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.480964 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.481099 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.481031 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.481682 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.481659 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.483538 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.483512 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.502461 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.502424 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5v2p\" (UniqueName: \"kubernetes.io/projected/91b6d2a6-3fce-4c5b-91e5-3353442a6243-kube-api-access-d5v2p\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.669221 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.669143 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:43.751589 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.751552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:43.970754 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:43.970711 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s"] Oct 31 01:23:43.973577 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:23:43.973546 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b6d2a6_3fce_4c5b_91e5_3353442a6243.slice/crio-e04dbedc9f09b5302e93fd359437eb3e44de5e90dbfd8d37fa401f569a195bc3 WatchSource:0}: Error finding container e04dbedc9f09b5302e93fd359437eb3e44de5e90dbfd8d37fa401f569a195bc3: Status 404 returned error can't find the container with id e04dbedc9f09b5302e93fd359437eb3e44de5e90dbfd8d37fa401f569a195bc3 Oct 31 01:23:44.191462 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:44.191417 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" event={"ID":"91b6d2a6-3fce-4c5b-91e5-3353442a6243","Type":"ContainerStarted","Data":"e04dbedc9f09b5302e93fd359437eb3e44de5e90dbfd8d37fa401f569a195bc3"} Oct 31 01:23:44.353392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:44.353351 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:44.498312 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:44.498227 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:45.027119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:45.027083 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:45.198513 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:45.198474 5978 generic.go:356] "Generic (PLEG): container finished" podID="91b6d2a6-3fce-4c5b-91e5-3353442a6243" containerID="6e0c8c0c06c4302bb47852918c2d017c571799c19dedec49b5a1421fc46b0e13" exitCode=0 Oct 31 01:23:45.198744 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:45.198552 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" event={"ID":"91b6d2a6-3fce-4c5b-91e5-3353442a6243","Type":"ContainerDied","Data":"6e0c8c0c06c4302bb47852918c2d017c571799c19dedec49b5a1421fc46b0e13"} Oct 31 01:23:45.773890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:45.773852 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:45.971164 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:45.971125 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:46.137282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.137238 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:46.307816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.307717 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:46.364656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.364630 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:46.415266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.415175 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-openshift-ca\") pod \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " Oct 31 01:23:46.415266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.415224 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b6d2a6-3fce-4c5b-91e5-3353442a6243-kube-api-access-d5v2p\" (UniqueName: \"kubernetes.io/projected/91b6d2a6-3fce-4c5b-91e5-3353442a6243-kube-api-access-d5v2p\") pod \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " Oct 31 01:23:46.415266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.415259 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-xks-ca\") pod \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\" (UID: \"91b6d2a6-3fce-4c5b-91e5-3353442a6243\") " Oct 31 01:23:46.415725 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.415701 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-openshift-ca" pod "91b6d2a6-3fce-4c5b-91e5-3353442a6243" (UID: "91b6d2a6-3fce-4c5b-91e5-3353442a6243"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:46.418252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.418194 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-xks-ca" pod "91b6d2a6-3fce-4c5b-91e5-3353442a6243" (UID: "91b6d2a6-3fce-4c5b-91e5-3353442a6243"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:46.418428 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.418367 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b6d2a6-3fce-4c5b-91e5-3353442a6243-kube-api-access-d5v2p" pod "91b6d2a6-3fce-4c5b-91e5-3353442a6243" (UID: "91b6d2a6-3fce-4c5b-91e5-3353442a6243"). InnerVolumeSpecName "kube-api-access-d5v2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:46.516827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.516791 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:46.516827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.516823 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5v2p\" (UniqueName: \"kubernetes.io/projected/91b6d2a6-3fce-4c5b-91e5-3353442a6243-kube-api-access-d5v2p\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:46.516827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.516835 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b6d2a6-3fce-4c5b-91e5-3353442a6243-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:46.985128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:46.985088 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:47.212526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:47.212501 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" Oct 31 01:23:47.212710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:47.212491 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s" event={"ID":"91b6d2a6-3fce-4c5b-91e5-3353442a6243","Type":"ContainerDied","Data":"e04dbedc9f09b5302e93fd359437eb3e44de5e90dbfd8d37fa401f569a195bc3"} Oct 31 01:23:47.212710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:47.212639 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e04dbedc9f09b5302e93fd359437eb3e44de5e90dbfd8d37fa401f569a195bc3" Oct 31 01:23:47.285731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:47.285642 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:47.479047 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:47.479008 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:47.686584 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:47.686518 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:48.053392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.053366 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:23:48.090037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.090006 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:23:48.097421 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.097391 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:48.111857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.111828 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:23:48.134324 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134291 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-socket\") pod \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " Oct 31 01:23:48.134582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134342 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-envoy\") pod \"0f68f517-8050-41a3-8dc3-690f0a9cb036\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " Oct 31 01:23:48.134582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134369 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-data\") pod \"0f68f517-8050-41a3-8dc3-690f0a9cb036\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " Oct 31 01:23:48.134582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134409 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-kube-api-access-n2m86\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-kube-api-access-n2m86\") pod \"0f68f517-8050-41a3-8dc3-690f0a9cb036\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " Oct 31 01:23:48.134582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134437 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-certs\") pod \"0f68f517-8050-41a3-8dc3-690f0a9cb036\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " Oct 31 01:23:48.134582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134533 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-socket\") pod \"0f68f517-8050-41a3-8dc3-690f0a9cb036\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134585 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-credential-socket\") pod \"b3feb663-8df2-41f4-acfa-4f883edb3740\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134616 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-podinfo\") pod \"b3feb663-8df2-41f4-acfa-4f883edb3740\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134641 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-envoy\") pod \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134669 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-socket" pod "71ed5649-6ad5-4423-8784-0e2fc1ed64bc" (UID: "71ed5649-6ad5-4423-8784-0e2fc1ed64bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134680 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-token\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-token\") pod \"0f68f517-8050-41a3-8dc3-690f0a9cb036\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134710 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-podinfo\") pod \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134759 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-data" pod "0f68f517-8050-41a3-8dc3-690f0a9cb036" (UID: "0f68f517-8050-41a3-8dc3-690f0a9cb036"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134782 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3feb663-8df2-41f4-acfa-4f883edb3740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3feb663-8df2-41f4-acfa-4f883edb3740-istiod-ca-cert\") pod \"b3feb663-8df2-41f4-acfa-4f883edb3740\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134818 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-kube-api-access-jnhvk\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-kube-api-access-jnhvk\") pod \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134853 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-token\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-token\") pod \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " Oct 31 01:23:48.134893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134896 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-credential-socket\") pod \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134930 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-kube-api-access-q4qsb\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-kube-api-access-q4qsb\") pod \"b3feb663-8df2-41f4-acfa-4f883edb3740\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.134959 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-socket" pod "0f68f517-8050-41a3-8dc3-690f0a9cb036" (UID: "0f68f517-8050-41a3-8dc3-690f0a9cb036"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135022 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-token\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-token\") pod \"b3feb663-8df2-41f4-acfa-4f883edb3740\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135060 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-certs\") pod \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135116 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-credential-socket\") pod \"0f68f517-8050-41a3-8dc3-690f0a9cb036\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135158 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-certs\") pod \"b3feb663-8df2-41f4-acfa-4f883edb3740\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135202 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-data\") pod \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135230 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-envoy\") pod \"b3feb663-8df2-41f4-acfa-4f883edb3740\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135266 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-podinfo\") pod \"0f68f517-8050-41a3-8dc3-690f0a9cb036\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135282 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-credential-socket" pod "b3feb663-8df2-41f4-acfa-4f883edb3740" (UID: "b3feb663-8df2-41f4-acfa-4f883edb3740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135302 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f68f517-8050-41a3-8dc3-690f0a9cb036-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f68f517-8050-41a3-8dc3-690f0a9cb036-istiod-ca-cert\") pod \"0f68f517-8050-41a3-8dc3-690f0a9cb036\" (UID: \"0f68f517-8050-41a3-8dc3-690f0a9cb036\") " Oct 31 01:23:48.135381 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135347 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-socket\") pod \"b3feb663-8df2-41f4-acfa-4f883edb3740\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " Oct 31 01:23:48.136151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135392 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istiod-ca-cert\") pod \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\" (UID: \"71ed5649-6ad5-4423-8784-0e2fc1ed64bc\") " Oct 31 01:23:48.136151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135425 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-data\") pod \"b3feb663-8df2-41f4-acfa-4f883edb3740\" (UID: \"b3feb663-8df2-41f4-acfa-4f883edb3740\") " Oct 31 01:23:48.136151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135432 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-certs" pod "0f68f517-8050-41a3-8dc3-690f0a9cb036" (UID: "0f68f517-8050-41a3-8dc3-690f0a9cb036"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.136151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.135977 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.136151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.136002 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.136151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.136018 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.136151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.136032 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.136151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.136046 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.137331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.136399 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-data" pod "b3feb663-8df2-41f4-acfa-4f883edb3740" (UID: "b3feb663-8df2-41f4-acfa-4f883edb3740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.137331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.136669 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-credential-socket" pod "0f68f517-8050-41a3-8dc3-690f0a9cb036" (UID: "0f68f517-8050-41a3-8dc3-690f0a9cb036"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.137331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.136898 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-certs" pod "71ed5649-6ad5-4423-8784-0e2fc1ed64bc" (UID: "71ed5649-6ad5-4423-8784-0e2fc1ed64bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.137331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.137082 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-certs" pod "b3feb663-8df2-41f4-acfa-4f883edb3740" (UID: "b3feb663-8df2-41f4-acfa-4f883edb3740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.137331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.137285 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-data" pod "71ed5649-6ad5-4423-8784-0e2fc1ed64bc" (UID: "71ed5649-6ad5-4423-8784-0e2fc1ed64bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.137604 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.137519 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-credential-socket" pod "71ed5649-6ad5-4423-8784-0e2fc1ed64bc" (UID: "71ed5649-6ad5-4423-8784-0e2fc1ed64bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.137726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.137698 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-socket" pod "b3feb663-8df2-41f4-acfa-4f883edb3740" (UID: "b3feb663-8df2-41f4-acfa-4f883edb3740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.137941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.137921 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f68f517-8050-41a3-8dc3-690f0a9cb036-istiod-ca-cert" pod "0f68f517-8050-41a3-8dc3-690f0a9cb036" (UID: "0f68f517-8050-41a3-8dc3-690f0a9cb036"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:48.138019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.137980 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3feb663-8df2-41f4-acfa-4f883edb3740-istiod-ca-cert" pod "b3feb663-8df2-41f4-acfa-4f883edb3740" (UID: "b3feb663-8df2-41f4-acfa-4f883edb3740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:48.138019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.137987 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-envoy" pod "71ed5649-6ad5-4423-8784-0e2fc1ed64bc" (UID: "71ed5649-6ad5-4423-8784-0e2fc1ed64bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.138158 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.138068 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-token" pod "b3feb663-8df2-41f4-acfa-4f883edb3740" (UID: "b3feb663-8df2-41f4-acfa-4f883edb3740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.138601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.138443 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istiod-ca-cert" pod "71ed5649-6ad5-4423-8784-0e2fc1ed64bc" (UID: "71ed5649-6ad5-4423-8784-0e2fc1ed64bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:48.138709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.138606 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-podinfo" pod "b3feb663-8df2-41f4-acfa-4f883edb3740" (UID: "b3feb663-8df2-41f4-acfa-4f883edb3740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:48.139306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.139284 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-envoy" pod "0f68f517-8050-41a3-8dc3-690f0a9cb036" (UID: "0f68f517-8050-41a3-8dc3-690f0a9cb036"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.139695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.139654 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-kube-api-access-n2m86" pod "0f68f517-8050-41a3-8dc3-690f0a9cb036" (UID: "0f68f517-8050-41a3-8dc3-690f0a9cb036"). InnerVolumeSpecName "kube-api-access-n2m86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.140154 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.140129 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-envoy" pod "b3feb663-8df2-41f4-acfa-4f883edb3740" (UID: "b3feb663-8df2-41f4-acfa-4f883edb3740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:23:48.140336 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.140308 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-kube-api-access-jnhvk" pod "71ed5649-6ad5-4423-8784-0e2fc1ed64bc" (UID: "71ed5649-6ad5-4423-8784-0e2fc1ed64bc"). InnerVolumeSpecName "kube-api-access-jnhvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.140444 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.140414 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-kube-api-access-q4qsb" pod "b3feb663-8df2-41f4-acfa-4f883edb3740" (UID: "b3feb663-8df2-41f4-acfa-4f883edb3740"). InnerVolumeSpecName "kube-api-access-q4qsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.140623 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.140601 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-podinfo" pod "0f68f517-8050-41a3-8dc3-690f0a9cb036" (UID: "0f68f517-8050-41a3-8dc3-690f0a9cb036"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:48.140916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.140896 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-token" pod "0f68f517-8050-41a3-8dc3-690f0a9cb036" (UID: "0f68f517-8050-41a3-8dc3-690f0a9cb036"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.141714 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.141682 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-token" pod "71ed5649-6ad5-4423-8784-0e2fc1ed64bc" (UID: "71ed5649-6ad5-4423-8784-0e2fc1ed64bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:48.142086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.142066 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-podinfo" pod "71ed5649-6ad5-4423-8784-0e2fc1ed64bc" (UID: "71ed5649-6ad5-4423-8784-0e2fc1ed64bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:23:48.223434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.223113 5978 generic.go:356] "Generic (PLEG): container finished" podID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerID="1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90" exitCode=0 Oct 31 01:23:48.223434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.223145 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" event={"ID":"b3feb663-8df2-41f4-acfa-4f883edb3740","Type":"ContainerDied","Data":"1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90"} Oct 31 01:23:48.223434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.223198 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" Oct 31 01:23:48.223434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.223219 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb" event={"ID":"b3feb663-8df2-41f4-acfa-4f883edb3740","Type":"ContainerDied","Data":"58a49ccb62300eb887e40cbd9a70de12c41423b6350aa02df8ce36217af45314"} Oct 31 01:23:48.223434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.223251 5978 scope.go:122] "RemoveContainer" containerID="1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90" Oct 31 01:23:48.225295 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.225270 5978 generic.go:356] "Generic (PLEG): container finished" podID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerID="fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee" exitCode=0 Oct 31 01:23:48.225432 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.225340 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" Oct 31 01:23:48.225432 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.225345 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" event={"ID":"71ed5649-6ad5-4423-8784-0e2fc1ed64bc","Type":"ContainerDied","Data":"fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee"} Oct 31 01:23:48.225432 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.225388 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc" event={"ID":"71ed5649-6ad5-4423-8784-0e2fc1ed64bc","Type":"ContainerDied","Data":"3dc77fc5d1878efe071af289bbf07e67130c993f2c4cae704f09f96fcf4f50b3"} Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.234885 5978 generic.go:356] "Generic (PLEG): container finished" podID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerID="4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675" exitCode=0 Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.235069 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" event={"ID":"0f68f517-8050-41a3-8dc3-690f0a9cb036","Type":"ContainerDied","Data":"4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675"} Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.235125 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" event={"ID":"0f68f517-8050-41a3-8dc3-690f0a9cb036","Type":"ContainerDied","Data":"0486d0f77482f63c63b70b48fd03fee38d1e507a3a8fd6c15782abbaa2f4d882"} Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.235214 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237512 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237536 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237569 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3feb663-8df2-41f4-acfa-4f883edb3740-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237584 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnhvk\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-kube-api-access-jnhvk\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237601 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237615 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237644 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4qsb\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-kube-api-access-q4qsb\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237658 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237671 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237684 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237712 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237726 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237740 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237753 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237766 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f68f517-8050-41a3-8dc3-690f0a9cb036-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237796 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237810 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237825 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237838 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f68f517-8050-41a3-8dc3-690f0a9cb036-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237862 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2m86\" (UniqueName: \"kubernetes.io/projected/0f68f517-8050-41a3-8dc3-690f0a9cb036-kube-api-access-n2m86\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237875 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3feb663-8df2-41f4-acfa-4f883edb3740-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.237890 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ed5649-6ad5-4423-8784-0e2fc1ed64bc-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:23:48.247101 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.238874 5978 scope.go:122] "RemoveContainer" containerID="c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591" Oct 31 01:23:48.254563 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.254530 5978 scope.go:122] "RemoveContainer" containerID="1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90" Oct 31 01:23:48.254976 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:48.254938 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90\": container with ID starting with 1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90 not found: ID does not exist" containerID="1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90" Oct 31 01:23:48.255070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.255004 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90"} err="failed to get container status \"1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90\": rpc error: code = NotFound desc = could not find container \"1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90\": container with ID starting with 1ba737448915d05ff822589e14f35ed8ed5bf149cbaff4a780d623eaeb24da90 not found: ID does not exist" Oct 31 01:23:48.255070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.255034 5978 scope.go:122] "RemoveContainer" containerID="c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591" Oct 31 01:23:48.255368 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:48.255349 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591\": container with ID starting with c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591 not found: ID does not exist" containerID="c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591" Oct 31 01:23:48.255415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.255378 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591"} err="failed to get container status \"c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591\": rpc error: code = NotFound desc = could not find container \"c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591\": container with ID starting with c262688d1b099dc0cb2de7d365732437c83dd0c80484a46f1544362846715591 not found: ID does not exist" Oct 31 01:23:48.255415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.255393 5978 scope.go:122] "RemoveContainer" containerID="fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee" Oct 31 01:23:48.267629 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.267608 5978 scope.go:122] "RemoveContainer" containerID="fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee" Oct 31 01:23:48.268005 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:48.267983 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee\": container with ID starting with fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee not found: ID does not exist" containerID="fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee" Oct 31 01:23:48.268056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.268017 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee"} err="failed to get container status \"fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee\": rpc error: code = NotFound desc = could not find container \"fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee\": container with ID starting with fada2a2e68f49c9a4a5623fd59ee68b9151f1892e0305f0434c4178aaa5ab6ee not found: ID does not exist" Oct 31 01:23:48.268056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.268038 5978 scope.go:122] "RemoveContainer" containerID="4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675" Oct 31 01:23:48.280714 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.280608 5978 scope.go:122] "RemoveContainer" containerID="4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675" Oct 31 01:23:48.281149 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:23:48.281120 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675\": container with ID starting with 4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675 not found: ID does not exist" containerID="4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675" Oct 31 01:23:48.281264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.281155 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675"} err="failed to get container status \"4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675\": rpc error: code = NotFound desc = could not find container \"4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675\": container with ID starting with 4ec072171e7729896c7f382fd87c4d6423e196dc8df435cf0bd511e887bc8675 not found: ID does not exist" Oct 31 01:23:48.294214 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.294177 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s"] Oct 31 01:23:48.298189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.298162 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-vhs2s"] Oct 31 01:23:48.328410 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.328369 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc"] Oct 31 01:23:48.337710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.337680 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-ntkbc"] Oct 31 01:23:48.368180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.368139 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb"] Oct 31 01:23:48.371184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.370568 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8f0029-openshift-default-dd4bb5c6c-pb5cb"] Oct 31 01:23:48.425562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.425528 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" path="/var/lib/kubelet/pods/0f68f517-8050-41a3-8dc3-690f0a9cb036/volumes" Oct 31 01:23:48.426005 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.425992 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" path="/var/lib/kubelet/pods/71ed5649-6ad5-4423-8784-0e2fc1ed64bc/volumes" Oct 31 01:23:48.426388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:48.426376 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" path="/var/lib/kubelet/pods/b3feb663-8df2-41f4-acfa-4f883edb3740/volumes" Oct 31 01:23:49.293553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:49.293510 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:49.783828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:49.783786 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:50.920814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:50.920778 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:51.599651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:51.599613 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:52.247130 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:52.247084 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:52.388849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:52.388808 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:52.607940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:52.607907 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:53.121081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:53.121040 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:53.493906 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:53.493809 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:53.946845 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:53.946809 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:54.165546 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:54.165508 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:55.156479 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:55.155923 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:55.679356 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:55.679311 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:55.752890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:55.752859 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s"] Oct 31 01:23:55.762351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:55.762322 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-36af31-trm6s"] Oct 31 01:23:55.823088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:55.823050 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:56.026494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:56.026373 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:56.292207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:56.292118 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:56.424691 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:56.424654 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b6d2a6-3fce-4c5b-91e5-3353442a6243" path="/var/lib/kubelet/pods/91b6d2a6-3fce-4c5b-91e5-3353442a6243/volumes" Oct 31 01:23:57.715892 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:57.715852 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:58.208497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:58.208437 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:59.299635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:23:59.299593 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:00.909303 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:00.909253 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:02.590501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:02.590467 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:02.739405 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:02.739365 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:03.032043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:03.031954 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf"] Oct 31 01:24:03.032217 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:03.032191 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" containerID="cri-o://61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50" gracePeriod=30 Oct 31 01:24:03.110957 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:03.110920 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:03.180494 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:24:03.180427 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:24:04.270208 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:04.270170 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph"] Oct 31 01:24:04.424582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:04.424540 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:04.574360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:04.574329 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:06.017971 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.017931 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87"] Oct 31 01:24:06.019047 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019016 5978 cpu_manager.go:414] "containerMap: removing container" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerName="istio-proxy" Oct 31 01:24:06.019047 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019043 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019069 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019075 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019087 5978 cpu_manager.go:414] "containerMap: removing container" podUID="91b6d2a6-3fce-4c5b-91e5-3353442a6243" containerName="revoke-keys" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019092 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b6d2a6-3fce-4c5b-91e5-3353442a6243" containerName="revoke-keys" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019101 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019106 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019128 5978 cpu_manager.go:414] "containerMap: removing container" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019132 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019198 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019214 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f68f517-8050-41a3-8dc3-690f0a9cb036" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019224 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ed5649-6ad5-4423-8784-0e2fc1ed64bc" containerName="istio-proxy" Oct 31 01:24:06.019254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.019232 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b6d2a6-3fce-4c5b-91e5-3353442a6243" containerName="revoke-keys" Oct 31 01:24:06.024498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.024476 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.036780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.036749 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:24:06.046330 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.046292 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:24:06.049211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.049182 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87"] Oct 31 01:24:06.129438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.129399 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28aabcff-gjn87\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.129650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.129575 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28aabcff-gjn87\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.129692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.129652 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxpb\" (UniqueName: \"kubernetes.io/projected/9ef1d20e-e378-4dd5-9570-6ce1207b3629-kube-api-access-jcxpb\") pod \"maas-api-revoke-sub-28aabcff-gjn87\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.231229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.231190 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28aabcff-gjn87\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.231426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.231245 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxpb\" (UniqueName: \"kubernetes.io/projected/9ef1d20e-e378-4dd5-9570-6ce1207b3629-kube-api-access-jcxpb\") pod \"maas-api-revoke-sub-28aabcff-gjn87\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.231426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.231314 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28aabcff-gjn87\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.232024 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.232003 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28aabcff-gjn87\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.233791 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.233767 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28aabcff-gjn87\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.252914 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.252881 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxpb\" (UniqueName: \"kubernetes.io/projected/9ef1d20e-e378-4dd5-9570-6ce1207b3629-kube-api-access-jcxpb\") pod \"maas-api-revoke-sub-28aabcff-gjn87\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.339477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.339434 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:06.494853 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.494806 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:06.667726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:06.667628 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87"] Oct 31 01:24:06.671901 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:24:06.671871 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef1d20e_e378_4dd5_9570_6ce1207b3629.slice/crio-291e16b177007ed944ad350fcd9b9651e30b8a0a01ca64da15e70a846b826f71 WatchSource:0}: Error finding container 291e16b177007ed944ad350fcd9b9651e30b8a0a01ca64da15e70a846b826f71: Status 404 returned error can't find the container with id 291e16b177007ed944ad350fcd9b9651e30b8a0a01ca64da15e70a846b826f71 Oct 31 01:24:07.355890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:07.355850 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" event={"ID":"9ef1d20e-e378-4dd5-9570-6ce1207b3629","Type":"ContainerStarted","Data":"a60cec6cffea7d0bf52fa3fc43ea4bb772a0bbbbdc8e0ff5bc8c5fbec6126134"} Oct 31 01:24:07.356388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:07.355934 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" event={"ID":"9ef1d20e-e378-4dd5-9570-6ce1207b3629","Type":"ContainerStarted","Data":"291e16b177007ed944ad350fcd9b9651e30b8a0a01ca64da15e70a846b826f71"} Oct 31 01:24:07.392186 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:07.392124 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" podStartSLOduration=2.392107403 podStartE2EDuration="2.392107403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:07.390771352 +0000 UTC m=+3865.088885099" watchObservedRunningTime="2026-10-31 01:24:07.392107403 +0000 UTC m=+3865.090221225" Oct 31 01:24:08.204071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.204033 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:08.294356 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.294324 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 01:24:08.365306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.365267 5978 generic.go:356] "Generic (PLEG): container finished" podID="395c04b2-9440-4612-8711-5566faf2c7c3" containerID="61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50" exitCode=0 Oct 31 01:24:08.365829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.365319 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" event={"ID":"395c04b2-9440-4612-8711-5566faf2c7c3","Type":"ContainerDied","Data":"61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50"} Oct 31 01:24:08.365829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.365377 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" event={"ID":"395c04b2-9440-4612-8711-5566faf2c7c3","Type":"ContainerDied","Data":"f500d1428d76fd6f580bbbd8371928bc455d9f6f60284a028dcab8fe5ef69df1"} Oct 31 01:24:08.365829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.365406 5978 scope.go:122] "RemoveContainer" containerID="61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50" Oct 31 01:24:08.366676 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.366650 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf" Oct 31 01:24:08.368349 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.368321 5978 generic.go:356] "Generic (PLEG): container finished" podID="9ef1d20e-e378-4dd5-9570-6ce1207b3629" containerID="a60cec6cffea7d0bf52fa3fc43ea4bb772a0bbbbdc8e0ff5bc8c5fbec6126134" exitCode=0 Oct 31 01:24:08.368476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.368403 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" event={"ID":"9ef1d20e-e378-4dd5-9570-6ce1207b3629","Type":"ContainerDied","Data":"a60cec6cffea7d0bf52fa3fc43ea4bb772a0bbbbdc8e0ff5bc8c5fbec6126134"} Oct 31 01:24:08.379170 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.379144 5978 scope.go:122] "RemoveContainer" containerID="61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50" Oct 31 01:24:08.379526 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:24:08.379493 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50\": container with ID starting with 61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50 not found: ID does not exist" containerID="61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50" Oct 31 01:24:08.379650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.379537 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50"} err="failed to get container status \"61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50\": rpc error: code = NotFound desc = could not find container \"61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50\": container with ID starting with 61358742817d081d5654e6efd8ebcd0db3e5d0ab9f9762ec68bd94a28ac3ff50 not found: ID does not exist" Oct 31 01:24:08.456207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.456170 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-kube-api-access-4k442\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-kube-api-access-4k442\") pod \"395c04b2-9440-4612-8711-5566faf2c7c3\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " Oct 31 01:24:08.456390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.456214 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-data\") pod \"395c04b2-9440-4612-8711-5566faf2c7c3\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " Oct 31 01:24:08.456390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.456237 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-istio-token\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-istio-token\") pod \"395c04b2-9440-4612-8711-5566faf2c7c3\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " Oct 31 01:24:08.456390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.456326 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-envoy\") pod \"395c04b2-9440-4612-8711-5566faf2c7c3\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " Oct 31 01:24:08.456390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.456373 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/395c04b2-9440-4612-8711-5566faf2c7c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/395c04b2-9440-4612-8711-5566faf2c7c3-istio-podinfo\") pod \"395c04b2-9440-4612-8711-5566faf2c7c3\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " Oct 31 01:24:08.456588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.456475 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-credential-socket\") pod \"395c04b2-9440-4612-8711-5566faf2c7c3\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " Oct 31 01:24:08.456588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.456515 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-certs\") pod \"395c04b2-9440-4612-8711-5566faf2c7c3\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " Oct 31 01:24:08.456588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.456550 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-socket\") pod \"395c04b2-9440-4612-8711-5566faf2c7c3\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " Oct 31 01:24:08.456715 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.456630 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/395c04b2-9440-4612-8711-5566faf2c7c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/395c04b2-9440-4612-8711-5566faf2c7c3-istiod-ca-cert\") pod \"395c04b2-9440-4612-8711-5566faf2c7c3\" (UID: \"395c04b2-9440-4612-8711-5566faf2c7c3\") " Oct 31 01:24:08.457289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.457257 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-data" pod "395c04b2-9440-4612-8711-5566faf2c7c3" (UID: "395c04b2-9440-4612-8711-5566faf2c7c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.457431 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.457345 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-credential-socket" pod "395c04b2-9440-4612-8711-5566faf2c7c3" (UID: "395c04b2-9440-4612-8711-5566faf2c7c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.457737 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.457714 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395c04b2-9440-4612-8711-5566faf2c7c3-istiod-ca-cert" pod "395c04b2-9440-4612-8711-5566faf2c7c3" (UID: "395c04b2-9440-4612-8711-5566faf2c7c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:08.457871 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.457771 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-socket" pod "395c04b2-9440-4612-8711-5566faf2c7c3" (UID: "395c04b2-9440-4612-8711-5566faf2c7c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.457871 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.457820 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-certs" pod "395c04b2-9440-4612-8711-5566faf2c7c3" (UID: "395c04b2-9440-4612-8711-5566faf2c7c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.459632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.459606 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-envoy" pod "395c04b2-9440-4612-8711-5566faf2c7c3" (UID: "395c04b2-9440-4612-8711-5566faf2c7c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:08.459721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.459673 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/395c04b2-9440-4612-8711-5566faf2c7c3-istio-podinfo" pod "395c04b2-9440-4612-8711-5566faf2c7c3" (UID: "395c04b2-9440-4612-8711-5566faf2c7c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:24:08.459764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.459711 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-kube-api-access-4k442" pod "395c04b2-9440-4612-8711-5566faf2c7c3" (UID: "395c04b2-9440-4612-8711-5566faf2c7c3"). InnerVolumeSpecName "kube-api-access-4k442". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:08.459801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.459760 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-istio-token" pod "395c04b2-9440-4612-8711-5566faf2c7c3" (UID: "395c04b2-9440-4612-8711-5566faf2c7c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:08.558448 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.558409 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.558448 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.558443 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.558694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.558485 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.558694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.558495 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/395c04b2-9440-4612-8711-5566faf2c7c3-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.558694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.558506 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k442\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-kube-api-access-4k442\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.558694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.558515 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.558694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.558523 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/395c04b2-9440-4612-8711-5566faf2c7c3-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.558694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.558530 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/395c04b2-9440-4612-8711-5566faf2c7c3-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.558694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.558547 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/395c04b2-9440-4612-8711-5566faf2c7c3-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:08.707487 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.707422 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf"] Oct 31 01:24:08.716901 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:08.716870 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-36af31-openshift-default-74f58f595f-zkrnf"] Oct 31 01:24:09.270516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.270436 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" podUID="c69bca4f-dbbf-404d-9c2e-401300cee4b5" containerName="payload-pre-processing" containerID="cri-o://30bac281023632bbc61597532466cdd0f54de1b88623da7014b3c1b9e7883e19" gracePeriod=25 Oct 31 01:24:09.512776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.512744 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:09.670678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.670631 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-openshift-ca\") pod \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " Oct 31 01:24:09.670894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.670821 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef1d20e-e378-4dd5-9570-6ce1207b3629-kube-api-access-jcxpb\" (UniqueName: \"kubernetes.io/projected/9ef1d20e-e378-4dd5-9570-6ce1207b3629-kube-api-access-jcxpb\") pod \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " Oct 31 01:24:09.670894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.670876 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-xks-ca\") pod \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\" (UID: \"9ef1d20e-e378-4dd5-9570-6ce1207b3629\") " Oct 31 01:24:09.671030 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.670973 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-openshift-ca" pod "9ef1d20e-e378-4dd5-9570-6ce1207b3629" (UID: "9ef1d20e-e378-4dd5-9570-6ce1207b3629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:09.671228 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.671201 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:09.672974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.672950 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef1d20e-e378-4dd5-9570-6ce1207b3629-kube-api-access-jcxpb" pod "9ef1d20e-e378-4dd5-9570-6ce1207b3629" (UID: "9ef1d20e-e378-4dd5-9570-6ce1207b3629"). InnerVolumeSpecName "kube-api-access-jcxpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:09.673082 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.673006 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-xks-ca" pod "9ef1d20e-e378-4dd5-9570-6ce1207b3629" (UID: "9ef1d20e-e378-4dd5-9570-6ce1207b3629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:09.772611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.772570 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcxpb\" (UniqueName: \"kubernetes.io/projected/9ef1d20e-e378-4dd5-9570-6ce1207b3629-kube-api-access-jcxpb\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:09.772611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.772603 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef1d20e-e378-4dd5-9570-6ce1207b3629-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:09.783896 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:09.783839 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:10.381973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:10.381940 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" Oct 31 01:24:10.382276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:10.381937 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87" event={"ID":"9ef1d20e-e378-4dd5-9570-6ce1207b3629","Type":"ContainerDied","Data":"291e16b177007ed944ad350fcd9b9651e30b8a0a01ca64da15e70a846b826f71"} Oct 31 01:24:10.382389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:10.382301 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="291e16b177007ed944ad350fcd9b9651e30b8a0a01ca64da15e70a846b826f71" Oct 31 01:24:10.424441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:10.424402 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" path="/var/lib/kubelet/pods/395c04b2-9440-4612-8711-5566faf2c7c3/volumes" Oct 31 01:24:10.764074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:10.763984 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" podUID="c69bca4f-dbbf-404d-9c2e-401300cee4b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:24:11.477392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:11.477339 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:12.995769 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:12.995728 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:14.135025 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.134986 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf"] Oct 31 01:24:14.167152 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.167110 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25"] Oct 31 01:24:14.232767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.232731 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl"] Oct 31 01:24:14.233044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.233005 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" podUID="8b3bf388-5757-464e-a5a9-9d7378c295c0" containerName="maas-api" containerID="cri-o://c3d896f526c2d197e7e6af1de3014c20b886c59e339c862244a7c89e514148b1" gracePeriod=30 Oct 31 01:24:14.315605 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.315566 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:14.409288 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.409127 5978 generic.go:356] "Generic (PLEG): container finished" podID="8b3bf388-5757-464e-a5a9-9d7378c295c0" containerID="c3d896f526c2d197e7e6af1de3014c20b886c59e339c862244a7c89e514148b1" exitCode=0 Oct 31 01:24:14.409288 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.409206 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" event={"ID":"8b3bf388-5757-464e-a5a9-9d7378c295c0","Type":"ContainerDied","Data":"c3d896f526c2d197e7e6af1de3014c20b886c59e339c862244a7c89e514148b1"} Oct 31 01:24:14.458945 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.458914 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:14.480076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.480054 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 01:24:14.516624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.516593 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-metrics-certs\") pod \"8b3bf388-5757-464e-a5a9-9d7378c295c0\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " Oct 31 01:24:14.516822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.516699 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3bf388-5757-464e-a5a9-9d7378c295c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3bf388-5757-464e-a5a9-9d7378c295c0-openshift-service-ca\") pod \"8b3bf388-5757-464e-a5a9-9d7378c295c0\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " Oct 31 01:24:14.516822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.516749 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3bf388-5757-464e-a5a9-9d7378c295c0-kube-api-access-qn4jr\" (UniqueName: \"kubernetes.io/projected/8b3bf388-5757-464e-a5a9-9d7378c295c0-kube-api-access-qn4jr\") pod \"8b3bf388-5757-464e-a5a9-9d7378c295c0\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " Oct 31 01:24:14.516822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.516802 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-maas-api-tls\") pod \"8b3bf388-5757-464e-a5a9-9d7378c295c0\" (UID: \"8b3bf388-5757-464e-a5a9-9d7378c295c0\") " Oct 31 01:24:14.517168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.517136 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3bf388-5757-464e-a5a9-9d7378c295c0-openshift-service-ca" pod "8b3bf388-5757-464e-a5a9-9d7378c295c0" (UID: "8b3bf388-5757-464e-a5a9-9d7378c295c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:14.519335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.519303 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-metrics-certs" pod "8b3bf388-5757-464e-a5a9-9d7378c295c0" (UID: "8b3bf388-5757-464e-a5a9-9d7378c295c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:14.520543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.520417 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-maas-api-tls" pod "8b3bf388-5757-464e-a5a9-9d7378c295c0" (UID: "8b3bf388-5757-464e-a5a9-9d7378c295c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:14.527240 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.527029 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3bf388-5757-464e-a5a9-9d7378c295c0-kube-api-access-qn4jr" pod "8b3bf388-5757-464e-a5a9-9d7378c295c0" (UID: "8b3bf388-5757-464e-a5a9-9d7378c295c0"). InnerVolumeSpecName "kube-api-access-qn4jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:14.618497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.618433 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:14.618497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.618484 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3bf388-5757-464e-a5a9-9d7378c295c0-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:14.618497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.618494 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3bf388-5757-464e-a5a9-9d7378c295c0-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:14.618497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:14.618506 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4jr\" (UniqueName: \"kubernetes.io/projected/8b3bf388-5757-464e-a5a9-9d7378c295c0-kube-api-access-qn4jr\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:15.002282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:15.002242 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:15.416300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:15.416271 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" Oct 31 01:24:15.416792 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:15.416270 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl" event={"ID":"8b3bf388-5757-464e-a5a9-9d7378c295c0","Type":"ContainerDied","Data":"55bed8881fe8756e82e97b4b40326dad7e33d056daea1f655594193d8e343563"} Oct 31 01:24:15.416792 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:15.416408 5978 scope.go:122] "RemoveContainer" containerID="c3d896f526c2d197e7e6af1de3014c20b886c59e339c862244a7c89e514148b1" Oct 31 01:24:15.464929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:15.464885 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl"] Oct 31 01:24:15.474729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:15.474697 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2bdf3-55dd9cf46-f4pzl"] Oct 31 01:24:15.566027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:15.565992 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:15.725410 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:15.725315 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:16.245388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:16.245343 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:16.426209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:16.426173 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3bf388-5757-464e-a5a9-9d7378c295c0" path="/var/lib/kubelet/pods/8b3bf388-5757-464e-a5a9-9d7378c295c0/volumes" Oct 31 01:24:16.915809 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:16.915765 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:18.460173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:18.460134 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:18.665424 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:18.665386 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:19.002279 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:19.002238 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:19.136260 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:19.136187 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" podUID="a99d5479-9fc7-4cbd-a242-9f2c33cead38" containerName="payload-processing" containerID="cri-o://3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7" gracePeriod=25 Oct 31 01:24:19.168337 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:19.168290 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" podUID="b85f151b-d6cf-4189-839b-1c55e2980350" containerName="payload-pre-processing" containerID="cri-o://ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1" gracePeriod=25 Oct 31 01:24:19.206054 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:19.206014 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:19.794988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:19.794949 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:20.763903 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:20.763869 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" podUID="c69bca4f-dbbf-404d-9c2e-401300cee4b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:24:21.064893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:21.064814 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:21.275159 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:21.275123 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:22.934698 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:22.934660 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:23.409916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:23.409882 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:23.575755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:23.575716 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:23.765659 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:23.765570 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:24.285202 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.285169 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:24.572860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.572829 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4"] Oct 31 01:24:24.573510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573431 5978 cpu_manager.go:414] "containerMap: removing container" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" Oct 31 01:24:24.573686 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573664 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" Oct 31 01:24:24.573786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573729 5978 cpu_manager.go:414] "containerMap: removing container" podUID="9ef1d20e-e378-4dd5-9570-6ce1207b3629" containerName="revoke-keys" Oct 31 01:24:24.573786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573739 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef1d20e-e378-4dd5-9570-6ce1207b3629" containerName="revoke-keys" Oct 31 01:24:24.573786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573755 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8b3bf388-5757-464e-a5a9-9d7378c295c0" containerName="maas-api" Oct 31 01:24:24.573786 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573763 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3bf388-5757-464e-a5a9-9d7378c295c0" containerName="maas-api" Oct 31 01:24:24.573996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573920 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3feb663-8df2-41f4-acfa-4f883edb3740" containerName="istio-proxy" Oct 31 01:24:24.573996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573940 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef1d20e-e378-4dd5-9570-6ce1207b3629" containerName="revoke-keys" Oct 31 01:24:24.573996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573952 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3bf388-5757-464e-a5a9-9d7378c295c0" containerName="maas-api" Oct 31 01:24:24.573996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.573968 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="395c04b2-9440-4612-8711-5566faf2c7c3" containerName="istio-proxy" Oct 31 01:24:24.577339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.577321 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.589905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.589873 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:24:24.589905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.589893 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:24:24.618474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.605285 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4"] Oct 31 01:24:24.618474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.605428 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:24:24.618474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.605521 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:24:24.627287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.627250 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncl22\" (UniqueName: \"kubernetes.io/projected/7756a5f3-eaf3-43aa-a8fb-89261a30e142-kube-api-access-ncl22\") pod \"maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.627519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.627395 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.627519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.627497 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.728507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.728476 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncl22\" (UniqueName: \"kubernetes.io/projected/7756a5f3-eaf3-43aa-a8fb-89261a30e142-kube-api-access-ncl22\") pod \"maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.728706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.728535 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.728706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.728567 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.729201 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.729177 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.730941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.730921 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.749159 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.749131 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncl22\" (UniqueName: \"kubernetes.io/projected/7756a5f3-eaf3-43aa-a8fb-89261a30e142-kube-api-access-ncl22\") pod \"maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:24.880639 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.880552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:24.891468 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:24.891432 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:25.186503 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:25.186432 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4"] Oct 31 01:24:25.189718 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:24:25.189690 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7756a5f3_eaf3_43aa_a8fb_89261a30e142.slice/crio-d71ee7a4c204c0ac6c8936f7fdaeced2017c3954f1a342595c93436ec583ebb3 WatchSource:0}: Error finding container d71ee7a4c204c0ac6c8936f7fdaeced2017c3954f1a342595c93436ec583ebb3: Status 404 returned error can't find the container with id d71ee7a4c204c0ac6c8936f7fdaeced2017c3954f1a342595c93436ec583ebb3 Oct 31 01:24:25.477421 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:25.477330 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" event={"ID":"7756a5f3-eaf3-43aa-a8fb-89261a30e142","Type":"ContainerStarted","Data":"d71ee7a4c204c0ac6c8936f7fdaeced2017c3954f1a342595c93436ec583ebb3"} Oct 31 01:24:26.054360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:26.054320 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" podUID="a99d5479-9fc7-4cbd-a242-9f2c33cead38" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:24:26.054579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:26.054401 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" podUID="b85f151b-d6cf-4189-839b-1c55e2980350" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:24:26.240511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:26.240475 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:26.386988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:26.386950 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:26.484504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:26.484425 5978 generic.go:356] "Generic (PLEG): container finished" podID="7756a5f3-eaf3-43aa-a8fb-89261a30e142" containerID="81ba9e730b576c487be4b0c49377700c63b5363693f8ff3418238ac9fb5f34a9" exitCode=0 Oct 31 01:24:26.484896 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:26.484522 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" event={"ID":"7756a5f3-eaf3-43aa-a8fb-89261a30e142","Type":"ContainerDied","Data":"81ba9e730b576c487be4b0c49377700c63b5363693f8ff3418238ac9fb5f34a9"} Oct 31 01:24:26.592415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:26.592377 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:26.734614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:26.734517 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:27.629442 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.629419 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:27.762118 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.762026 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-xks-ca\") pod \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " Oct 31 01:24:27.762310 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.762134 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-openshift-ca\") pod \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " Oct 31 01:24:27.762310 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.762265 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7756a5f3-eaf3-43aa-a8fb-89261a30e142-kube-api-access-ncl22\" (UniqueName: \"kubernetes.io/projected/7756a5f3-eaf3-43aa-a8fb-89261a30e142-kube-api-access-ncl22\") pod \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\" (UID: \"7756a5f3-eaf3-43aa-a8fb-89261a30e142\") " Oct 31 01:24:27.762492 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.762468 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-openshift-ca" pod "7756a5f3-eaf3-43aa-a8fb-89261a30e142" (UID: "7756a5f3-eaf3-43aa-a8fb-89261a30e142"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:27.762656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.762638 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:27.764262 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.764240 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7756a5f3-eaf3-43aa-a8fb-89261a30e142-kube-api-access-ncl22" pod "7756a5f3-eaf3-43aa-a8fb-89261a30e142" (UID: "7756a5f3-eaf3-43aa-a8fb-89261a30e142"). InnerVolumeSpecName "kube-api-access-ncl22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:27.764262 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.764254 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-xks-ca" pod "7756a5f3-eaf3-43aa-a8fb-89261a30e142" (UID: "7756a5f3-eaf3-43aa-a8fb-89261a30e142"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:27.863360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.863325 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncl22\" (UniqueName: \"kubernetes.io/projected/7756a5f3-eaf3-43aa-a8fb-89261a30e142-kube-api-access-ncl22\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:27.863360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.863359 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7756a5f3-eaf3-43aa-a8fb-89261a30e142-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:27.914781 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:27.914745 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:28.201911 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:28.201876 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:28.496355 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:28.496274 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" Oct 31 01:24:28.496355 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:28.496313 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4" event={"ID":"7756a5f3-eaf3-43aa-a8fb-89261a30e142","Type":"ContainerDied","Data":"d71ee7a4c204c0ac6c8936f7fdaeced2017c3954f1a342595c93436ec583ebb3"} Oct 31 01:24:28.496567 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:28.496363 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d71ee7a4c204c0ac6c8936f7fdaeced2017c3954f1a342595c93436ec583ebb3" Oct 31 01:24:29.186934 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.186895 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc"] Oct 31 01:24:29.196639 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.196606 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj"] Oct 31 01:24:29.349607 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.349577 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:29.503699 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.503668 5978 generic.go:356] "Generic (PLEG): container finished" podID="c69bca4f-dbbf-404d-9c2e-401300cee4b5" containerID="30bac281023632bbc61597532466cdd0f54de1b88623da7014b3c1b9e7883e19" exitCode=0 Oct 31 01:24:29.503849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.503724 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" event={"ID":"c69bca4f-dbbf-404d-9c2e-401300cee4b5","Type":"ContainerDied","Data":"30bac281023632bbc61597532466cdd0f54de1b88623da7014b3c1b9e7883e19"} Oct 31 01:24:29.526856 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.526826 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 01:24:29.579933 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.579896 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69bca4f-dbbf-404d-9c2e-401300cee4b5-kube-api-access-ws5kh\" (UniqueName: \"kubernetes.io/projected/c69bca4f-dbbf-404d-9c2e-401300cee4b5-kube-api-access-ws5kh\") pod \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\" (UID: \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\") " Oct 31 01:24:29.580143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.580022 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69bca4f-dbbf-404d-9c2e-401300cee4b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bca4f-dbbf-404d-9c2e-401300cee4b5-plugins-config-volume\") pod \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\" (UID: \"c69bca4f-dbbf-404d-9c2e-401300cee4b5\") " Oct 31 01:24:29.580435 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.580407 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69bca4f-dbbf-404d-9c2e-401300cee4b5-plugins-config-volume" pod "c69bca4f-dbbf-404d-9c2e-401300cee4b5" (UID: "c69bca4f-dbbf-404d-9c2e-401300cee4b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:29.581988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.581956 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69bca4f-dbbf-404d-9c2e-401300cee4b5-kube-api-access-ws5kh" pod "c69bca4f-dbbf-404d-9c2e-401300cee4b5" (UID: "c69bca4f-dbbf-404d-9c2e-401300cee4b5"). InnerVolumeSpecName "kube-api-access-ws5kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:29.681547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.681515 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws5kh\" (UniqueName: \"kubernetes.io/projected/c69bca4f-dbbf-404d-9c2e-401300cee4b5-kube-api-access-ws5kh\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:29.681547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.681545 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bca4f-dbbf-404d-9c2e-401300cee4b5-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:29.895093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:29.895054 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:30.510037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:30.510005 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" Oct 31 01:24:30.510037 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:30.510021 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph" event={"ID":"c69bca4f-dbbf-404d-9c2e-401300cee4b5","Type":"ContainerDied","Data":"e76217e58452df51989250e3f2f354ee7f303ce47fca704dc50afc65b4c1399f"} Oct 31 01:24:30.510582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:30.510093 5978 scope.go:122] "RemoveContainer" containerID="30bac281023632bbc61597532466cdd0f54de1b88623da7014b3c1b9e7883e19" Oct 31 01:24:30.556210 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:30.556172 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph"] Oct 31 01:24:30.560348 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:30.560323 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c9e377-67f55d5979-774ph"] Oct 31 01:24:31.525085 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:31.525042 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:32.424669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:32.424639 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69bca4f-dbbf-404d-9c2e-401300cee4b5" path="/var/lib/kubelet/pods/c69bca4f-dbbf-404d-9c2e-401300cee4b5/volumes" Oct 31 01:24:33.182056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:33.182019 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:33.182639 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:24:33.182609 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:24:34.790813 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:34.790771 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:35.097007 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:35.096970 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w"] Oct 31 01:24:35.129586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:35.129545 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5"] Oct 31 01:24:35.276879 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:35.276842 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:35.420046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:35.419953 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:36.054219 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:36.054183 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" podUID="b85f151b-d6cf-4189-839b-1c55e2980350" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:24:36.055511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:36.054981 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" podUID="a99d5479-9fc7-4cbd-a242-9f2c33cead38" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:24:36.399832 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:36.399797 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:37.653433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.653397 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn"] Oct 31 01:24:37.654334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.654001 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c69bca4f-dbbf-404d-9c2e-401300cee4b5" containerName="payload-pre-processing" Oct 31 01:24:37.654334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.654018 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69bca4f-dbbf-404d-9c2e-401300cee4b5" containerName="payload-pre-processing" Oct 31 01:24:37.654334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.654029 5978 cpu_manager.go:414] "containerMap: removing container" podUID="7756a5f3-eaf3-43aa-a8fb-89261a30e142" containerName="revoke-keys" Oct 31 01:24:37.654334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.654036 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7756a5f3-eaf3-43aa-a8fb-89261a30e142" containerName="revoke-keys" Oct 31 01:24:37.654334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.654129 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69bca4f-dbbf-404d-9c2e-401300cee4b5" containerName="payload-pre-processing" Oct 31 01:24:37.654334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.654144 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="7756a5f3-eaf3-43aa-a8fb-89261a30e142" containerName="revoke-keys" Oct 31 01:24:37.658911 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.658892 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.674248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.674226 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:24:37.675104 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.675087 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:24:37.684152 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.684119 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:24:37.684310 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.684181 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:24:37.687077 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.687053 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn"] Oct 31 01:24:37.766504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.766433 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d4260a-g9xsn\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.766695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.766515 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v457\" (UniqueName: \"kubernetes.io/projected/46d8d172-d451-41de-bf5d-fd5af6482b03-kube-api-access-7v457\") pod \"maas-api-revoke-sub-99d4260a-g9xsn\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.766695 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.766636 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d4260a-g9xsn\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.867860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.867816 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d4260a-g9xsn\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.868073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.867960 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d4260a-g9xsn\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.868073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.868005 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v457\" (UniqueName: \"kubernetes.io/projected/46d8d172-d451-41de-bf5d-fd5af6482b03-kube-api-access-7v457\") pod \"maas-api-revoke-sub-99d4260a-g9xsn\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.868503 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.868485 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d4260a-g9xsn\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.870395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.870372 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d4260a-g9xsn\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.889521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.889489 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v457\" (UniqueName: \"kubernetes.io/projected/46d8d172-d451-41de-bf5d-fd5af6482b03-kube-api-access-7v457\") pod \"maas-api-revoke-sub-99d4260a-g9xsn\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:37.973060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:37.972984 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:38.269560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:38.269480 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn"] Oct 31 01:24:38.272288 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:24:38.272258 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d8d172_d451_41de_bf5d_fd5af6482b03.slice/crio-7c075191b54ea3673d4cf1e5435498cf5a2473fda4e3bbc567c0460ae4dff1e8 WatchSource:0}: Error finding container 7c075191b54ea3673d4cf1e5435498cf5a2473fda4e3bbc567c0460ae4dff1e8: Status 404 returned error can't find the container with id 7c075191b54ea3673d4cf1e5435498cf5a2473fda4e3bbc567c0460ae4dff1e8 Oct 31 01:24:38.555364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:38.555314 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" event={"ID":"46d8d172-d451-41de-bf5d-fd5af6482b03","Type":"ContainerStarted","Data":"687d720c978f7a700bbec562e9b67e0d5e7da962e8df5602b9631dbabe846c9b"} Oct 31 01:24:38.555597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:38.555388 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" event={"ID":"46d8d172-d451-41de-bf5d-fd5af6482b03","Type":"ContainerStarted","Data":"7c075191b54ea3673d4cf1e5435498cf5a2473fda4e3bbc567c0460ae4dff1e8"} Oct 31 01:24:38.617283 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:38.617224 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" podStartSLOduration=1.6172069169999999 podStartE2EDuration="1.617206917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:38.616797561 +0000 UTC m=+3896.314911326" watchObservedRunningTime="2026-10-31 01:24:38.617206917 +0000 UTC m=+3896.315320681" Oct 31 01:24:39.419442 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.419415 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 01:24:39.450863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.450834 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 01:24:39.561534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.561500 5978 generic.go:356] "Generic (PLEG): container finished" podID="a99d5479-9fc7-4cbd-a242-9f2c33cead38" containerID="3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7" exitCode=0 Oct 31 01:24:39.561730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.561550 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" Oct 31 01:24:39.561730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.561587 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" event={"ID":"a99d5479-9fc7-4cbd-a242-9f2c33cead38","Type":"ContainerDied","Data":"3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7"} Oct 31 01:24:39.561730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.561641 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf" event={"ID":"a99d5479-9fc7-4cbd-a242-9f2c33cead38","Type":"ContainerDied","Data":"7335d7bf44ee612b230f74f5b7033439f5d5d08846cfa5f8babd5b4dac66c3a5"} Oct 31 01:24:39.561730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.561670 5978 scope.go:122] "RemoveContainer" containerID="3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7" Oct 31 01:24:39.562945 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.562925 5978 generic.go:356] "Generic (PLEG): container finished" podID="b85f151b-d6cf-4189-839b-1c55e2980350" containerID="ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1" exitCode=0 Oct 31 01:24:39.563041 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.562955 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" event={"ID":"b85f151b-d6cf-4189-839b-1c55e2980350","Type":"ContainerDied","Data":"ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1"} Oct 31 01:24:39.563041 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.563007 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" event={"ID":"b85f151b-d6cf-4189-839b-1c55e2980350","Type":"ContainerDied","Data":"8655e3b0d32e7ca9ff9d07c533fa3aa0ccf0b4823c4ad2130c88e7d481be421b"} Oct 31 01:24:39.563041 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.562968 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25" Oct 31 01:24:39.564594 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.564569 5978 generic.go:356] "Generic (PLEG): container finished" podID="46d8d172-d451-41de-bf5d-fd5af6482b03" containerID="687d720c978f7a700bbec562e9b67e0d5e7da962e8df5602b9631dbabe846c9b" exitCode=0 Oct 31 01:24:39.564700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.564602 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" event={"ID":"46d8d172-d451-41de-bf5d-fd5af6482b03","Type":"ContainerDied","Data":"687d720c978f7a700bbec562e9b67e0d5e7da962e8df5602b9631dbabe846c9b"} Oct 31 01:24:39.571697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.571682 5978 scope.go:122] "RemoveContainer" containerID="3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7" Oct 31 01:24:39.571976 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:24:39.571957 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7\": container with ID starting with 3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7 not found: ID does not exist" containerID="3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7" Oct 31 01:24:39.572057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.571995 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7"} err="failed to get container status \"3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7\": rpc error: code = NotFound desc = could not find container \"3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7\": container with ID starting with 3c2748bece1ff92f3df1a3aa9f2a8b8c843bdb69d075d125bf901ba39e8bf9a7 not found: ID does not exist" Oct 31 01:24:39.572057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.572019 5978 scope.go:122] "RemoveContainer" containerID="ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1" Oct 31 01:24:39.580724 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.580699 5978 scope.go:122] "RemoveContainer" containerID="ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1" Oct 31 01:24:39.580941 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:24:39.580925 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1\": container with ID starting with ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1 not found: ID does not exist" containerID="ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1" Oct 31 01:24:39.580993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.580948 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1"} err="failed to get container status \"ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1\": rpc error: code = NotFound desc = could not find container \"ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1\": container with ID starting with ead6dde0590bd51f2a1a6e9a2cc49ecb91c3f6b850db82d10663aea0a1ebaad1 not found: ID does not exist" Oct 31 01:24:39.584334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.584296 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99d5479-9fc7-4cbd-a242-9f2c33cead38-kube-api-access-h4csm\" (UniqueName: \"kubernetes.io/projected/a99d5479-9fc7-4cbd-a242-9f2c33cead38-kube-api-access-h4csm\") pod \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\" (UID: \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\") " Oct 31 01:24:39.584399 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.584373 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85f151b-d6cf-4189-839b-1c55e2980350-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85f151b-d6cf-4189-839b-1c55e2980350-plugins-config-volume\") pod \"b85f151b-d6cf-4189-839b-1c55e2980350\" (UID: \"b85f151b-d6cf-4189-839b-1c55e2980350\") " Oct 31 01:24:39.584436 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.584408 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85f151b-d6cf-4189-839b-1c55e2980350-kube-api-access-8wh7l\" (UniqueName: \"kubernetes.io/projected/b85f151b-d6cf-4189-839b-1c55e2980350-kube-api-access-8wh7l\") pod \"b85f151b-d6cf-4189-839b-1c55e2980350\" (UID: \"b85f151b-d6cf-4189-839b-1c55e2980350\") " Oct 31 01:24:39.584497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.584466 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99d5479-9fc7-4cbd-a242-9f2c33cead38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d5479-9fc7-4cbd-a242-9f2c33cead38-plugins-config-volume\") pod \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\" (UID: \"a99d5479-9fc7-4cbd-a242-9f2c33cead38\") " Oct 31 01:24:39.584798 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.584775 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99d5479-9fc7-4cbd-a242-9f2c33cead38-plugins-config-volume" pod "a99d5479-9fc7-4cbd-a242-9f2c33cead38" (UID: "a99d5479-9fc7-4cbd-a242-9f2c33cead38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:39.584798 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.584789 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85f151b-d6cf-4189-839b-1c55e2980350-plugins-config-volume" pod "b85f151b-d6cf-4189-839b-1c55e2980350" (UID: "b85f151b-d6cf-4189-839b-1c55e2980350"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:39.586360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.586343 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99d5479-9fc7-4cbd-a242-9f2c33cead38-kube-api-access-h4csm" pod "a99d5479-9fc7-4cbd-a242-9f2c33cead38" (UID: "a99d5479-9fc7-4cbd-a242-9f2c33cead38"). InnerVolumeSpecName "kube-api-access-h4csm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:39.586442 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.586401 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85f151b-d6cf-4189-839b-1c55e2980350-kube-api-access-8wh7l" pod "b85f151b-d6cf-4189-839b-1c55e2980350" (UID: "b85f151b-d6cf-4189-839b-1c55e2980350"). InnerVolumeSpecName "kube-api-access-8wh7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:39.685748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.685717 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4csm\" (UniqueName: \"kubernetes.io/projected/a99d5479-9fc7-4cbd-a242-9f2c33cead38-kube-api-access-h4csm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:39.685748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.685745 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85f151b-d6cf-4189-839b-1c55e2980350-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:39.685748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.685754 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wh7l\" (UniqueName: \"kubernetes.io/projected/b85f151b-d6cf-4189-839b-1c55e2980350-kube-api-access-8wh7l\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:39.685981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.685763 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d5479-9fc7-4cbd-a242-9f2c33cead38-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:39.917635 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.917597 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25"] Oct 31 01:24:39.926817 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.926787 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f2bdf3-7bf45d675f-8pf25"] Oct 31 01:24:39.953168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.953132 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf"] Oct 31 01:24:39.958056 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:39.958029 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f2bdf3-68f945d689-pqxbf"] Oct 31 01:24:40.098576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.098509 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" podUID="415e9e8c-2289-468b-9d49-33498212dd0f" containerName="payload-processing" containerID="cri-o://80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2" gracePeriod=25 Oct 31 01:24:40.130313 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.130222 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" podUID="e21d5451-a5c3-4f7e-8fe0-fae74204a205" containerName="payload-pre-processing" containerID="cri-o://b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240" gracePeriod=25 Oct 31 01:24:40.369388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.369365 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 01:24:40.424632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.424541 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99d5479-9fc7-4cbd-a242-9f2c33cead38" path="/var/lib/kubelet/pods/a99d5479-9fc7-4cbd-a242-9f2c33cead38/volumes" Oct 31 01:24:40.425120 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.424997 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85f151b-d6cf-4189-839b-1c55e2980350" path="/var/lib/kubelet/pods/b85f151b-d6cf-4189-839b-1c55e2980350/volumes" Oct 31 01:24:40.494872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.494836 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415e9e8c-2289-468b-9d49-33498212dd0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e9e8c-2289-468b-9d49-33498212dd0f-plugins-config-volume\") pod \"415e9e8c-2289-468b-9d49-33498212dd0f\" (UID: \"415e9e8c-2289-468b-9d49-33498212dd0f\") " Oct 31 01:24:40.495068 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.495008 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415e9e8c-2289-468b-9d49-33498212dd0f-kube-api-access-2xwh2\" (UniqueName: \"kubernetes.io/projected/415e9e8c-2289-468b-9d49-33498212dd0f-kube-api-access-2xwh2\") pod \"415e9e8c-2289-468b-9d49-33498212dd0f\" (UID: \"415e9e8c-2289-468b-9d49-33498212dd0f\") " Oct 31 01:24:40.495275 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.495250 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415e9e8c-2289-468b-9d49-33498212dd0f-plugins-config-volume" pod "415e9e8c-2289-468b-9d49-33498212dd0f" (UID: "415e9e8c-2289-468b-9d49-33498212dd0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:40.495516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.495499 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e9e8c-2289-468b-9d49-33498212dd0f-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:40.497162 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.497136 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415e9e8c-2289-468b-9d49-33498212dd0f-kube-api-access-2xwh2" pod "415e9e8c-2289-468b-9d49-33498212dd0f" (UID: "415e9e8c-2289-468b-9d49-33498212dd0f"). InnerVolumeSpecName "kube-api-access-2xwh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:40.573751 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.573709 5978 generic.go:356] "Generic (PLEG): container finished" podID="415e9e8c-2289-468b-9d49-33498212dd0f" containerID="80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2" exitCode=0 Oct 31 01:24:40.573950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.573757 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" Oct 31 01:24:40.573950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.573794 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" event={"ID":"415e9e8c-2289-468b-9d49-33498212dd0f","Type":"ContainerDied","Data":"80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2"} Oct 31 01:24:40.573950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.573876 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w" event={"ID":"415e9e8c-2289-468b-9d49-33498212dd0f","Type":"ContainerDied","Data":"5fbd686fbe29c329a8014e1dc02acecd81fb70b39115e653a3c4e15faf6a1e03"} Oct 31 01:24:40.573950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.573913 5978 scope.go:122] "RemoveContainer" containerID="80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2" Oct 31 01:24:40.585615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.585562 5978 scope.go:122] "RemoveContainer" containerID="80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2" Oct 31 01:24:40.586030 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:24:40.586003 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2\": container with ID starting with 80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2 not found: ID does not exist" containerID="80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2" Oct 31 01:24:40.586145 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.586042 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2"} err="failed to get container status \"80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2\": rpc error: code = NotFound desc = could not find container \"80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2\": container with ID starting with 80f0a1bf62dd520b25ca698d668adbcba99485c23f749f2d7aee4c5588aadaa2 not found: ID does not exist" Oct 31 01:24:40.609492 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.601151 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwh2\" (UniqueName: \"kubernetes.io/projected/415e9e8c-2289-468b-9d49-33498212dd0f-kube-api-access-2xwh2\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:40.619034 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.618983 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w"] Oct 31 01:24:40.624124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.624099 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-735123-8466d8d68f-ssg4w"] Oct 31 01:24:40.709424 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.709395 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:40.803103 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.803063 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-xks-ca\") pod \"46d8d172-d451-41de-bf5d-fd5af6482b03\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " Oct 31 01:24:40.803304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.803263 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-openshift-ca\") pod \"46d8d172-d451-41de-bf5d-fd5af6482b03\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " Oct 31 01:24:40.803380 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.803313 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d8d172-d451-41de-bf5d-fd5af6482b03-kube-api-access-7v457\" (UniqueName: \"kubernetes.io/projected/46d8d172-d451-41de-bf5d-fd5af6482b03-kube-api-access-7v457\") pod \"46d8d172-d451-41de-bf5d-fd5af6482b03\" (UID: \"46d8d172-d451-41de-bf5d-fd5af6482b03\") " Oct 31 01:24:40.803688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.803656 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-openshift-ca" pod "46d8d172-d451-41de-bf5d-fd5af6482b03" (UID: "46d8d172-d451-41de-bf5d-fd5af6482b03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:40.805041 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.805017 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-xks-ca" pod "46d8d172-d451-41de-bf5d-fd5af6482b03" (UID: "46d8d172-d451-41de-bf5d-fd5af6482b03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:40.805337 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.805310 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d8d172-d451-41de-bf5d-fd5af6482b03-kube-api-access-7v457" pod "46d8d172-d451-41de-bf5d-fd5af6482b03" (UID: "46d8d172-d451-41de-bf5d-fd5af6482b03"). InnerVolumeSpecName "kube-api-access-7v457". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:40.904700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.904655 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:40.904700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.904687 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v457\" (UniqueName: \"kubernetes.io/projected/46d8d172-d451-41de-bf5d-fd5af6482b03-kube-api-access-7v457\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:40.904700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:40.904698 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d8d172-d451-41de-bf5d-fd5af6482b03-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:41.583304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:41.583270 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" Oct 31 01:24:41.583304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:41.583284 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn" event={"ID":"46d8d172-d451-41de-bf5d-fd5af6482b03","Type":"ContainerDied","Data":"7c075191b54ea3673d4cf1e5435498cf5a2473fda4e3bbc567c0460ae4dff1e8"} Oct 31 01:24:41.583854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:41.583348 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c075191b54ea3673d4cf1e5435498cf5a2473fda4e3bbc567c0460ae4dff1e8" Oct 31 01:24:42.424747 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:42.424715 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415e9e8c-2289-468b-9d49-33498212dd0f" path="/var/lib/kubelet/pods/415e9e8c-2289-468b-9d49-33498212dd0f/volumes" Oct 31 01:24:44.208216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.208175 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" containerID="cri-o://9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775" gracePeriod=45 Oct 31 01:24:44.219963 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.219926 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" containerID="cri-o://35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3" gracePeriod=45 Oct 31 01:24:44.469656 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.469594 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 01:24:44.487898 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.487867 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 01:24:44.610201 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.610164 5978 generic.go:356] "Generic (PLEG): container finished" podID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerID="35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3" exitCode=0 Oct 31 01:24:44.610390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.610233 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" Oct 31 01:24:44.610390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.610250 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" event={"ID":"eaa793a8-217f-4df2-986a-fcb54b191a46","Type":"ContainerDied","Data":"35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3"} Oct 31 01:24:44.610390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.610293 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj" event={"ID":"eaa793a8-217f-4df2-986a-fcb54b191a46","Type":"ContainerDied","Data":"ceebb569f409ae9af770daa073a6906a785d3d70e3d2c5c115ca4f74f3e7f558"} Oct 31 01:24:44.610390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.610315 5978 scope.go:122] "RemoveContainer" containerID="35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3" Oct 31 01:24:44.611710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.611690 5978 generic.go:356] "Generic (PLEG): container finished" podID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerID="9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775" exitCode=0 Oct 31 01:24:44.611830 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.611726 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" event={"ID":"efda8a17-abaa-4e88-ab91-3f376bdec9cb","Type":"ContainerDied","Data":"9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775"} Oct 31 01:24:44.611830 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.611755 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" event={"ID":"efda8a17-abaa-4e88-ab91-3f376bdec9cb","Type":"ContainerDied","Data":"630eed6004c90a0565c4559222be1fac05be7b8a1332aa742a54fefb5d9482ff"} Oct 31 01:24:44.611830 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.611730 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc" Oct 31 01:24:44.620554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.620526 5978 scope.go:122] "RemoveContainer" containerID="35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3" Oct 31 01:24:44.620882 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:24:44.620854 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3\": container with ID starting with 35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3 not found: ID does not exist" containerID="35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3" Oct 31 01:24:44.620984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.620890 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3"} err="failed to get container status \"35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3\": rpc error: code = NotFound desc = could not find container \"35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3\": container with ID starting with 35f52bca040ef696960831c8bbbff34f56821f95f994c050a8cdc3d794c567d3 not found: ID does not exist" Oct 31 01:24:44.620984 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.620908 5978 scope.go:122] "RemoveContainer" containerID="9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775" Oct 31 01:24:44.630576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.630552 5978 scope.go:122] "RemoveContainer" containerID="9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775" Oct 31 01:24:44.630889 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:24:44.630871 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775\": container with ID starting with 9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775 not found: ID does not exist" containerID="9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775" Oct 31 01:24:44.630970 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.630896 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775"} err="failed to get container status \"9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775\": rpc error: code = NotFound desc = could not find container \"9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775\": container with ID starting with 9d6369ec05f9863ed08c39ba316f28d73c0ca83d3ad00fb456ca97c9987ae775 not found: ID does not exist" Oct 31 01:24:44.646386 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646357 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-tmp-dir\") pod \"eaa793a8-217f-4df2-986a-fcb54b191a46\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " Oct 31 01:24:44.646558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646413 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efda8a17-abaa-4e88-ab91-3f376bdec9cb-kube-api-access-sxgrj\" (UniqueName: \"kubernetes.io/projected/efda8a17-abaa-4e88-ab91-3f376bdec9cb-kube-api-access-sxgrj\") pod \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " Oct 31 01:24:44.646558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646487 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa793a8-217f-4df2-986a-fcb54b191a46-kube-api-access-6c6jq\" (UniqueName: \"kubernetes.io/projected/eaa793a8-217f-4df2-986a-fcb54b191a46-kube-api-access-6c6jq\") pod \"eaa793a8-217f-4df2-986a-fcb54b191a46\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " Oct 31 01:24:44.646694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646559 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaa793a8-217f-4df2-986a-fcb54b191a46-tls-certs\" (UniqueName: \"kubernetes.io/secret/eaa793a8-217f-4df2-986a-fcb54b191a46-tls-certs\") pod \"eaa793a8-217f-4df2-986a-fcb54b191a46\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " Oct 31 01:24:44.646694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646614 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tls-certs\") pod \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " Oct 31 01:24:44.646694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646655 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-home\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-home\") pod \"eaa793a8-217f-4df2-986a-fcb54b191a46\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " Oct 31 01:24:44.646694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646682 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-tmp-dir" pod "eaa793a8-217f-4df2-986a-fcb54b191a46" (UID: "eaa793a8-217f-4df2-986a-fcb54b191a46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.646909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646718 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-dshm\") pod \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " Oct 31 01:24:44.646909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646758 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-model-cache\") pod \"eaa793a8-217f-4df2-986a-fcb54b191a46\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " Oct 31 01:24:44.646909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646792 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tmp-dir\") pod \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " Oct 31 01:24:44.646909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646863 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-model-cache\") pod \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " Oct 31 01:24:44.646909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646893 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-home\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-home\") pod \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\" (UID: \"efda8a17-abaa-4e88-ab91-3f376bdec9cb\") " Oct 31 01:24:44.647235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646959 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-dshm\") pod \"eaa793a8-217f-4df2-986a-fcb54b191a46\" (UID: \"eaa793a8-217f-4df2-986a-fcb54b191a46\") " Oct 31 01:24:44.647235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.646980 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-home" pod "eaa793a8-217f-4df2-986a-fcb54b191a46" (UID: "eaa793a8-217f-4df2-986a-fcb54b191a46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.647235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.647082 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-model-cache" pod "eaa793a8-217f-4df2-986a-fcb54b191a46" (UID: "eaa793a8-217f-4df2-986a-fcb54b191a46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.647235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.647158 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-model-cache" pod "efda8a17-abaa-4e88-ab91-3f376bdec9cb" (UID: "efda8a17-abaa-4e88-ab91-3f376bdec9cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.647619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.647345 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-home" pod "efda8a17-abaa-4e88-ab91-3f376bdec9cb" (UID: "efda8a17-abaa-4e88-ab91-3f376bdec9cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.647619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.647358 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.647619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.647378 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.647619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.647392 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.647619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.647406 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.647619 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.647597 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tmp-dir" pod "efda8a17-abaa-4e88-ab91-3f376bdec9cb" (UID: "efda8a17-abaa-4e88-ab91-3f376bdec9cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.649060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.649040 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-dshm" pod "efda8a17-abaa-4e88-ab91-3f376bdec9cb" (UID: "efda8a17-abaa-4e88-ab91-3f376bdec9cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.649207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.649191 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tls-certs" pod "efda8a17-abaa-4e88-ab91-3f376bdec9cb" (UID: "efda8a17-abaa-4e88-ab91-3f376bdec9cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:44.649530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.649506 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efda8a17-abaa-4e88-ab91-3f376bdec9cb-kube-api-access-sxgrj" pod "efda8a17-abaa-4e88-ab91-3f376bdec9cb" (UID: "efda8a17-abaa-4e88-ab91-3f376bdec9cb"). InnerVolumeSpecName "kube-api-access-sxgrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:44.649735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.649714 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa793a8-217f-4df2-986a-fcb54b191a46-tls-certs" pod "eaa793a8-217f-4df2-986a-fcb54b191a46" (UID: "eaa793a8-217f-4df2-986a-fcb54b191a46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:44.649854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.649764 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-dshm" pod "eaa793a8-217f-4df2-986a-fcb54b191a46" (UID: "eaa793a8-217f-4df2-986a-fcb54b191a46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:44.649918 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.649904 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa793a8-217f-4df2-986a-fcb54b191a46-kube-api-access-6c6jq" pod "eaa793a8-217f-4df2-986a-fcb54b191a46" (UID: "eaa793a8-217f-4df2-986a-fcb54b191a46"). InnerVolumeSpecName "kube-api-access-6c6jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:44.747969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.747881 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaa793a8-217f-4df2-986a-fcb54b191a46-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.747969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.747911 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.747969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.747922 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.747969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.747932 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.747969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.747944 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efda8a17-abaa-4e88-ab91-3f376bdec9cb-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.747969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.747953 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaa793a8-217f-4df2-986a-fcb54b191a46-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.747969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.747966 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgrj\" (UniqueName: \"kubernetes.io/projected/efda8a17-abaa-4e88-ab91-3f376bdec9cb-kube-api-access-sxgrj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.748314 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.747981 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6jq\" (UniqueName: \"kubernetes.io/projected/eaa793a8-217f-4df2-986a-fcb54b191a46-kube-api-access-6c6jq\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:24:44.954161 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.954103 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj"] Oct 31 01:24:44.963476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.963431 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-w2-735123-kserve-6664f4f65b-mk4mj"] Oct 31 01:24:44.988800 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.988772 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc"] Oct 31 01:24:44.992892 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:44.992866 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-735123/distinct-2-w2-735123-kserve-54ffd9c447-mdflc"] Oct 31 01:24:46.423210 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:46.423176 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" path="/var/lib/kubelet/pods/eaa793a8-217f-4df2-986a-fcb54b191a46/volumes" Oct 31 01:24:46.423632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:46.423596 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" path="/var/lib/kubelet/pods/efda8a17-abaa-4e88-ab91-3f376bdec9cb/volumes" Oct 31 01:24:48.750003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:48.749957 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" podUID="e21d5451-a5c3-4f7e-8fe0-fae74204a205" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:24:50.688649 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:50.688619 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:24:50.696604 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:50.696580 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:24:50.958767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:50.958683 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:52.873675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:52.873635 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:53.321361 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:53.321331 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:53.395502 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:53.395462 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4"] Oct 31 01:24:53.405162 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:53.405125 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-735123-4bgt4"] Oct 31 01:24:53.493081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:53.493049 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:53.940866 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:53.940829 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:54.423708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:54.423673 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7756a5f3-eaf3-43aa-a8fb-89261a30e142" path="/var/lib/kubelet/pods/7756a5f3-eaf3-43aa-a8fb-89261a30e142/volumes" Oct 31 01:24:58.750674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:24:58.750631 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" podUID="e21d5451-a5c3-4f7e-8fe0-fae74204a205" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:25:00.261093 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.261051 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:00.385041 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.385011 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 01:25:00.426350 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.425859 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:00.491723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.491690 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21d5451-a5c3-4f7e-8fe0-fae74204a205-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21d5451-a5c3-4f7e-8fe0-fae74204a205-plugins-config-volume\") pod \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\" (UID: \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\") " Oct 31 01:25:00.491905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.491768 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21d5451-a5c3-4f7e-8fe0-fae74204a205-kube-api-access-r2nbs\" (UniqueName: \"kubernetes.io/projected/e21d5451-a5c3-4f7e-8fe0-fae74204a205-kube-api-access-r2nbs\") pod \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\" (UID: \"e21d5451-a5c3-4f7e-8fe0-fae74204a205\") " Oct 31 01:25:00.492073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.492051 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21d5451-a5c3-4f7e-8fe0-fae74204a205-plugins-config-volume" pod "e21d5451-a5c3-4f7e-8fe0-fae74204a205" (UID: "e21d5451-a5c3-4f7e-8fe0-fae74204a205"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:00.493829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.493802 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21d5451-a5c3-4f7e-8fe0-fae74204a205-kube-api-access-r2nbs" pod "e21d5451-a5c3-4f7e-8fe0-fae74204a205" (UID: "e21d5451-a5c3-4f7e-8fe0-fae74204a205"). InnerVolumeSpecName "kube-api-access-r2nbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:00.593122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.593081 5978 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21d5451-a5c3-4f7e-8fe0-fae74204a205-plugins-config-volume\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:25:00.593122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.593116 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2nbs\" (UniqueName: \"kubernetes.io/projected/e21d5451-a5c3-4f7e-8fe0-fae74204a205-kube-api-access-r2nbs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:25:00.632579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.632538 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:00.697634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.697540 5978 generic.go:356] "Generic (PLEG): container finished" podID="e21d5451-a5c3-4f7e-8fe0-fae74204a205" containerID="b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240" exitCode=0 Oct 31 01:25:00.697634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.697590 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" Oct 31 01:25:00.697885 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.697630 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" event={"ID":"e21d5451-a5c3-4f7e-8fe0-fae74204a205","Type":"ContainerDied","Data":"b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240"} Oct 31 01:25:00.697885 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.697686 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5" event={"ID":"e21d5451-a5c3-4f7e-8fe0-fae74204a205","Type":"ContainerDied","Data":"d5553919d6be470bf098c79576a1cfd1cc1107543acad91c99a36a8efa166d7c"} Oct 31 01:25:00.697885 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.697715 5978 scope.go:122] "RemoveContainer" containerID="b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240" Oct 31 01:25:00.707433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.707411 5978 scope.go:122] "RemoveContainer" containerID="b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240" Oct 31 01:25:00.707706 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:25:00.707687 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240\": container with ID starting with b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240 not found: ID does not exist" containerID="b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240" Oct 31 01:25:00.707758 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.707715 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240"} err="failed to get container status \"b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240\": rpc error: code = NotFound desc = could not find container \"b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240\": container with ID starting with b60259b13caafcbbabdd7de358cfad05fd0230118ceae987b45817c497af0240 not found: ID does not exist" Oct 31 01:25:00.747827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.747794 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5"] Oct 31 01:25:00.752556 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:00.752529 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-735123-6bd8bf8799-x9zr5"] Oct 31 01:25:02.187704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:02.187666 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:02.423749 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:02.423711 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21d5451-a5c3-4f7e-8fe0-fae74204a205" path="/var/lib/kubelet/pods/e21d5451-a5c3-4f7e-8fe0-fae74204a205/volumes" Oct 31 01:25:03.180750 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:25:03.180714 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:25:06.713815 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:06.713777 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:06.857796 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:06.857760 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:07.586342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:07.586305 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:13.538553 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:13.538506 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:25.082973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.082933 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn"] Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083495 5978 cpu_manager.go:414] "containerMap: removing container" podUID="46d8d172-d451-41de-bf5d-fd5af6482b03" containerName="revoke-keys" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083511 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d8d172-d451-41de-bf5d-fd5af6482b03" containerName="revoke-keys" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083526 5978 cpu_manager.go:414] "containerMap: removing container" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083534 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083544 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e21d5451-a5c3-4f7e-8fe0-fae74204a205" containerName="payload-pre-processing" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083549 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21d5451-a5c3-4f7e-8fe0-fae74204a205" containerName="payload-pre-processing" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083558 5978 cpu_manager.go:414] "containerMap: removing container" podUID="415e9e8c-2289-468b-9d49-33498212dd0f" containerName="payload-processing" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083563 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415e9e8c-2289-468b-9d49-33498212dd0f" containerName="payload-processing" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083574 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b85f151b-d6cf-4189-839b-1c55e2980350" containerName="payload-pre-processing" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083578 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85f151b-d6cf-4189-839b-1c55e2980350" containerName="payload-pre-processing" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083587 5978 cpu_manager.go:414] "containerMap: removing container" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083591 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083602 5978 cpu_manager.go:414] "containerMap: removing container" podUID="a99d5479-9fc7-4cbd-a242-9f2c33cead38" containerName="payload-processing" Oct 31 01:25:25.083665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083610 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99d5479-9fc7-4cbd-a242-9f2c33cead38" containerName="payload-processing" Oct 31 01:25:25.084150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083690 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21d5451-a5c3-4f7e-8fe0-fae74204a205" containerName="payload-pre-processing" Oct 31 01:25:25.084150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083701 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="415e9e8c-2289-468b-9d49-33498212dd0f" containerName="payload-processing" Oct 31 01:25:25.084150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083707 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85f151b-d6cf-4189-839b-1c55e2980350" containerName="payload-pre-processing" Oct 31 01:25:25.084150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083714 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa793a8-217f-4df2-986a-fcb54b191a46" containerName="main" Oct 31 01:25:25.084150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083719 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99d5479-9fc7-4cbd-a242-9f2c33cead38" containerName="payload-processing" Oct 31 01:25:25.084150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083730 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d8d172-d451-41de-bf5d-fd5af6482b03" containerName="revoke-keys" Oct 31 01:25:25.084150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.083737 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="efda8a17-abaa-4e88-ab91-3f376bdec9cb" containerName="main" Oct 31 01:25:25.086941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.086922 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.099294 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.099269 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:25:25.109103 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.109070 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:25:25.109103 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.109079 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:25:25.109103 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.109084 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:25:25.110914 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.110893 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn"] Oct 31 01:25:25.241678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.241639 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b89g2\" (UniqueName: \"kubernetes.io/projected/d1623e9a-d6d5-48fc-a93a-678d36aad715-kube-api-access-b89g2\") pod \"maas-api-revoke-sub-85237e45-k9cxn\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.241678 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.241674 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-k9cxn\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.241904 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.241814 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-k9cxn\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.343032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.342942 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b89g2\" (UniqueName: \"kubernetes.io/projected/d1623e9a-d6d5-48fc-a93a-678d36aad715-kube-api-access-b89g2\") pod \"maas-api-revoke-sub-85237e45-k9cxn\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.343032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.342988 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-k9cxn\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.343295 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.343061 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-k9cxn\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.343780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.343756 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-k9cxn\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.345393 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.345372 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-k9cxn\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.364749 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.364715 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b89g2\" (UniqueName: \"kubernetes.io/projected/d1623e9a-d6d5-48fc-a93a-678d36aad715-kube-api-access-b89g2\") pod \"maas-api-revoke-sub-85237e45-k9cxn\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.400410 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.400365 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:25.696227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.696187 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn"] Oct 31 01:25:25.699883 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:25:25.699836 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1623e9a_d6d5_48fc_a93a_678d36aad715.slice/crio-248a15b86bcba3da7b34c7a7e0c4bb5caa0c9c71dafaae17f1adcb9482903996 WatchSource:0}: Error finding container 248a15b86bcba3da7b34c7a7e0c4bb5caa0c9c71dafaae17f1adcb9482903996: Status 404 returned error can't find the container with id 248a15b86bcba3da7b34c7a7e0c4bb5caa0c9c71dafaae17f1adcb9482903996 Oct 31 01:25:25.828017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:25.827973 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" event={"ID":"d1623e9a-d6d5-48fc-a93a-678d36aad715","Type":"ContainerStarted","Data":"248a15b86bcba3da7b34c7a7e0c4bb5caa0c9c71dafaae17f1adcb9482903996"} Oct 31 01:25:26.835853 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:26.835817 5978 generic.go:356] "Generic (PLEG): container finished" podID="d1623e9a-d6d5-48fc-a93a-678d36aad715" containerID="c56529bd07f8996039168085a0f1f76e037880c1b540d382a7529c8ce0feed24" exitCode=0 Oct 31 01:25:26.836282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:26.835869 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" event={"ID":"d1623e9a-d6d5-48fc-a93a-678d36aad715","Type":"ContainerDied","Data":"c56529bd07f8996039168085a0f1f76e037880c1b540d382a7529c8ce0feed24"} Oct 31 01:25:27.972638 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:27.972606 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:28.065721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.065680 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-openshift-ca\") pod \"d1623e9a-d6d5-48fc-a93a-678d36aad715\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " Oct 31 01:25:28.065927 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.065748 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1623e9a-d6d5-48fc-a93a-678d36aad715-kube-api-access-b89g2\" (UniqueName: \"kubernetes.io/projected/d1623e9a-d6d5-48fc-a93a-678d36aad715-kube-api-access-b89g2\") pod \"d1623e9a-d6d5-48fc-a93a-678d36aad715\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " Oct 31 01:25:28.065927 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.065781 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-xks-ca\") pod \"d1623e9a-d6d5-48fc-a93a-678d36aad715\" (UID: \"d1623e9a-d6d5-48fc-a93a-678d36aad715\") " Oct 31 01:25:28.066064 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.066046 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-openshift-ca" pod "d1623e9a-d6d5-48fc-a93a-678d36aad715" (UID: "d1623e9a-d6d5-48fc-a93a-678d36aad715"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:28.067830 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.067807 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-xks-ca" pod "d1623e9a-d6d5-48fc-a93a-678d36aad715" (UID: "d1623e9a-d6d5-48fc-a93a-678d36aad715"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:28.067989 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.067970 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1623e9a-d6d5-48fc-a93a-678d36aad715-kube-api-access-b89g2" pod "d1623e9a-d6d5-48fc-a93a-678d36aad715" (UID: "d1623e9a-d6d5-48fc-a93a-678d36aad715"). InnerVolumeSpecName "kube-api-access-b89g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:28.166928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.166849 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:25:28.166928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.166878 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b89g2\" (UniqueName: \"kubernetes.io/projected/d1623e9a-d6d5-48fc-a93a-678d36aad715-kube-api-access-b89g2\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:25:28.166928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.166888 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1623e9a-d6d5-48fc-a93a-678d36aad715-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:25:28.848556 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.848522 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" event={"ID":"d1623e9a-d6d5-48fc-a93a-678d36aad715","Type":"ContainerDied","Data":"248a15b86bcba3da7b34c7a7e0c4bb5caa0c9c71dafaae17f1adcb9482903996"} Oct 31 01:25:28.848556 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.848549 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn" Oct 31 01:25:28.848774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:28.848570 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="248a15b86bcba3da7b34c7a7e0c4bb5caa0c9c71dafaae17f1adcb9482903996" Oct 31 01:25:33.183161 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:25:33.183114 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:25:35.622205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.622169 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84"] Oct 31 01:25:35.622730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.622696 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d1623e9a-d6d5-48fc-a93a-678d36aad715" containerName="revoke-keys" Oct 31 01:25:35.622730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.622716 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1623e9a-d6d5-48fc-a93a-678d36aad715" containerName="revoke-keys" Oct 31 01:25:35.622874 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.622859 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1623e9a-d6d5-48fc-a93a-678d36aad715" containerName="revoke-keys" Oct 31 01:25:35.626043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.626027 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.638988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.638962 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 01:25:35.639173 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.639060 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vtxt8\"" Oct 31 01:25:35.653702 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.653667 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84"] Oct 31 01:25:35.740191 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.740138 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.740387 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.740231 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.740387 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.740288 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.740387 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.740334 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8ccj\" (UniqueName: \"kubernetes.io/projected/95964f90-7a0d-4fe8-97aa-6f10cf619a81-kube-api-access-n8ccj\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.740387 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.740382 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.740614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.740411 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.740614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.740477 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.740614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.740504 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.740614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.740529 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.841362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841325 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.841582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841381 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ccj\" (UniqueName: \"kubernetes.io/projected/95964f90-7a0d-4fe8-97aa-6f10cf619a81-kube-api-access-n8ccj\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.841582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841402 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.841582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841437 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.841582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841473 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.841582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841496 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.841916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841599 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.841916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841702 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.841916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841757 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.842079 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.841989 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.842079 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.842042 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.842223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.842192 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.842286 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.842204 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.842343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.842291 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.844433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.844405 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.844652 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.844635 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.862108 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.862077 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95964f90-7a0d-4fe8-97aa-6f10cf619a81-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.862280 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.862161 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ccj\" (UniqueName: \"kubernetes.io/projected/95964f90-7a0d-4fe8-97aa-6f10cf619a81-kube-api-access-n8ccj\") pod \"maas-default-gateway-openshift-default-6886b96f9-ldf84\" (UID: \"95964f90-7a0d-4fe8-97aa-6f10cf619a81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:35.939704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:35.939614 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:36.250318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:36.250217 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84"] Oct 31 01:25:36.254545 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:25:36.254510 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95964f90_7a0d_4fe8_97aa_6f10cf619a81.slice/crio-0420cf990429da31d311d42ce2a49105baa49307e3a16180a1f957bc8a2408cd WatchSource:0}: Error finding container 0420cf990429da31d311d42ce2a49105baa49307e3a16180a1f957bc8a2408cd: Status 404 returned error can't find the container with id 0420cf990429da31d311d42ce2a49105baa49307e3a16180a1f957bc8a2408cd Oct 31 01:25:36.256690 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:36.256654 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:25:36.256765 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:36.256725 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:25:36.256765 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:36.256758 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:25:36.891120 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:36.891082 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" event={"ID":"95964f90-7a0d-4fe8-97aa-6f10cf619a81","Type":"ContainerStarted","Data":"9d737eba99b504258fd4056be8fb5ad52a39b23b466854b6ca93c610ee8aac92"} Oct 31 01:25:36.891616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:36.891135 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" event={"ID":"95964f90-7a0d-4fe8-97aa-6f10cf619a81","Type":"ContainerStarted","Data":"0420cf990429da31d311d42ce2a49105baa49307e3a16180a1f957bc8a2408cd"} Oct 31 01:25:36.940331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:36.940291 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:36.960680 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:36.960624 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" podStartSLOduration=1.960608355 podStartE2EDuration="1.960608355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:36.959579717 +0000 UTC m=+3954.657693482" watchObservedRunningTime="2026-10-31 01:25:36.960608355 +0000 UTC m=+3954.658722122" Oct 31 01:25:37.088167 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:37.088112 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:37.941908 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:37.941865 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ldf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 01:25:37.942394 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:37.941946 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" podUID="95964f90-7a0d-4fe8-97aa-6f10cf619a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 01:25:38.940560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:38.940522 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ldf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 01:25:38.940757 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:38.940589 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" podUID="95964f90-7a0d-4fe8-97aa-6f10cf619a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 01:25:39.633474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:39.633046 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:39.940071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:39.939977 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ldf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 01:25:39.940071 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:39.940042 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" podUID="95964f90-7a0d-4fe8-97aa-6f10cf619a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 01:25:40.941082 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:40.941045 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ldf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 01:25:40.941593 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:40.941111 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" podUID="95964f90-7a0d-4fe8-97aa-6f10cf619a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 01:25:41.687900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:41.687862 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:41.836959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:41.836918 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:41.940657 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:41.940573 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ldf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 01:25:41.940657 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:41.940627 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" podUID="95964f90-7a0d-4fe8-97aa-6f10cf619a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 01:25:42.940088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:42.940060 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ldf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 01:25:42.940516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:42.940122 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" podUID="95964f90-7a0d-4fe8-97aa-6f10cf619a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 01:25:44.564934 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:44.564899 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:44.565615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:44.565331 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:44.566442 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:44.566421 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" Oct 31 01:25:49.122372 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:49.122334 5978 scope.go:122] "RemoveContainer" containerID="e1032af286ba127378b54f5b8934c25abbc278329074dd293ddcbd62d7c527d6" Oct 31 01:25:50.665527 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.665493 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6"] Oct 31 01:25:50.669266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.669245 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.700675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.700645 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6"] Oct 31 01:25:50.800051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.800018 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.800051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.800058 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.800282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.800085 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.800282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.800229 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.800282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.800270 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.800404 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.800292 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jfjd\" (UniqueName: \"kubernetes.io/projected/694e2b84-5e34-4d1e-bf99-16b261a95272-kube-api-access-9jfjd\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.800404 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.800373 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.800404 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.800397 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.800535 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.800420 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694e2b84-5e34-4d1e-bf99-16b261a95272-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.901641 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.901597 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.901641 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.901643 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.901937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.901663 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jfjd\" (UniqueName: \"kubernetes.io/projected/694e2b84-5e34-4d1e-bf99-16b261a95272-kube-api-access-9jfjd\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.901937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.901688 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.901937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.901707 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.901937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.901734 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694e2b84-5e34-4d1e-bf99-16b261a95272-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.901937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.901820 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.901937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.901852 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.901937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.901886 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.902390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.902134 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.902390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.902188 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.902390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.902258 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.902390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.902271 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.902595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.902571 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694e2b84-5e34-4d1e-bf99-16b261a95272-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.904225 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.904204 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.904464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.904429 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.922760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.922695 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694e2b84-5e34-4d1e-bf99-16b261a95272-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.922982 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.922956 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jfjd\" (UniqueName: \"kubernetes.io/projected/694e2b84-5e34-4d1e-bf99-16b261a95272-kube-api-access-9jfjd\") pod \"maas-default-gateway-openshift-default-6886b96f9-6bxb6\" (UID: \"694e2b84-5e34-4d1e-bf99-16b261a95272\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:50.982558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:50.982516 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:51.028309 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.028272 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd"] Oct 31 01:25:51.030722 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.030693 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc9351a-d9tzd"] Oct 31 01:25:51.100872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.100829 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:51.237637 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.237544 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:51.294822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.294792 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6"] Oct 31 01:25:51.297094 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:25:51.297064 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694e2b84_5e34_4d1e_bf99_16b261a95272.slice/crio-b785f528de5916a34e980972a8d6410fca67f59283a6f9c81ef41291dc493660 WatchSource:0}: Error finding container b785f528de5916a34e980972a8d6410fca67f59283a6f9c81ef41291dc493660: Status 404 returned error can't find the container with id b785f528de5916a34e980972a8d6410fca67f59283a6f9c81ef41291dc493660 Oct 31 01:25:51.299300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.299268 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:25:51.299367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.299337 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:25:51.299414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.299365 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:25:51.992069 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.992032 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" event={"ID":"694e2b84-5e34-4d1e-bf99-16b261a95272","Type":"ContainerStarted","Data":"b8aa831f4d086a5ff4a4c4fc08d6b17bd69f72bcf244328427ff17f49e5dac93"} Oct 31 01:25:51.992488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:51.992084 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" event={"ID":"694e2b84-5e34-4d1e-bf99-16b261a95272","Type":"ContainerStarted","Data":"b785f528de5916a34e980972a8d6410fca67f59283a6f9c81ef41291dc493660"} Oct 31 01:25:52.043177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:52.043120 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" podStartSLOduration=2.043106461 podStartE2EDuration="2.043106461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:52.042106587 +0000 UTC m=+3969.740220352" watchObservedRunningTime="2026-10-31 01:25:52.043106461 +0000 UTC m=+3969.741220226" Oct 31 01:25:52.441299 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:52.441263 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8352c968-6c31-496d-8ae9-b8c66e31fce0" path="/var/lib/kubelet/pods/8352c968-6c31-496d-8ae9-b8c66e31fce0/volumes" Oct 31 01:25:52.983248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:52.983204 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:52.984971 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:52.984939 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6bxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 01:25:52.985123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:52.985004 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" podUID="694e2b84-5e34-4d1e-bf99-16b261a95272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 01:25:53.983371 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:53.983280 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6bxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 01:25:53.983371 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:53.983334 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" podUID="694e2b84-5e34-4d1e-bf99-16b261a95272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 01:25:54.983548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:54.983513 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6bxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 01:25:54.984081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:54.983575 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" podUID="694e2b84-5e34-4d1e-bf99-16b261a95272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 01:25:55.983219 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:55.983182 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6bxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 01:25:55.983422 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:55.983243 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" podUID="694e2b84-5e34-4d1e-bf99-16b261a95272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 01:25:56.983841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:56.983808 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6bxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 01:25:56.984334 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:56.984304 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" podUID="694e2b84-5e34-4d1e-bf99-16b261a95272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 01:25:57.983778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:57.983733 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6bxb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 01:25:57.983980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:57.983824 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" podUID="694e2b84-5e34-4d1e-bf99-16b261a95272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 01:25:59.558252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:59.558213 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:59.558788 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:59.558618 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:25:59.559488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:25:59.559447 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" Oct 31 01:26:03.180862 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:26:03.180824 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:26:27.355607 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.355569 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv"] Oct 31 01:26:27.359344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.359319 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.371586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.371560 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:26:27.381091 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.381064 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:26:27.381290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.381074 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:26:27.381364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.381099 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:26:27.383442 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.383416 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv"] Oct 31 01:26:27.500989 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.500947 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sb85\" (UniqueName: \"kubernetes.io/projected/740a1a3b-f68f-438f-ba0b-225b5c71f798-kube-api-access-8sb85\") pod \"maas-api-revoke-sub-c1e271a9-pmqlv\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.501188 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.501067 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pmqlv\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.501188 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.501109 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pmqlv\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.602206 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.602164 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pmqlv\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.602402 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.602224 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pmqlv\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.602402 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.602309 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sb85\" (UniqueName: \"kubernetes.io/projected/740a1a3b-f68f-438f-ba0b-225b5c71f798-kube-api-access-8sb85\") pod \"maas-api-revoke-sub-c1e271a9-pmqlv\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.603031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.603006 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pmqlv\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.604834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.604809 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pmqlv\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.623562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.623490 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sb85\" (UniqueName: \"kubernetes.io/projected/740a1a3b-f68f-438f-ba0b-225b5c71f798-kube-api-access-8sb85\") pod \"maas-api-revoke-sub-c1e271a9-pmqlv\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.674241 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.674200 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:27.968513 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:27.968409 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv"] Oct 31 01:26:27.972030 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:26:27.971996 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740a1a3b_f68f_438f_ba0b_225b5c71f798.slice/crio-99d4b89aacaf417822ad61503294316c85c63d1abe1b595b68e14cc029170f5d WatchSource:0}: Error finding container 99d4b89aacaf417822ad61503294316c85c63d1abe1b595b68e14cc029170f5d: Status 404 returned error can't find the container with id 99d4b89aacaf417822ad61503294316c85c63d1abe1b595b68e14cc029170f5d Oct 31 01:26:28.214814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:28.214778 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" event={"ID":"740a1a3b-f68f-438f-ba0b-225b5c71f798","Type":"ContainerStarted","Data":"99d4b89aacaf417822ad61503294316c85c63d1abe1b595b68e14cc029170f5d"} Oct 31 01:26:29.223615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:29.223581 5978 generic.go:356] "Generic (PLEG): container finished" podID="740a1a3b-f68f-438f-ba0b-225b5c71f798" containerID="769c79c0fc7bf6e1bc3e73bbfa28966a29a161a1867d0aa077f0d309836ef895" exitCode=0 Oct 31 01:26:29.224041 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:29.223663 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" event={"ID":"740a1a3b-f68f-438f-ba0b-225b5c71f798","Type":"ContainerDied","Data":"769c79c0fc7bf6e1bc3e73bbfa28966a29a161a1867d0aa077f0d309836ef895"} Oct 31 01:26:30.377119 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.377091 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:30.530902 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.530814 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-openshift-ca\") pod \"740a1a3b-f68f-438f-ba0b-225b5c71f798\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " Oct 31 01:26:30.531059 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.530940 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-xks-ca\") pod \"740a1a3b-f68f-438f-ba0b-225b5c71f798\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " Oct 31 01:26:30.531059 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.530996 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740a1a3b-f68f-438f-ba0b-225b5c71f798-kube-api-access-8sb85\" (UniqueName: \"kubernetes.io/projected/740a1a3b-f68f-438f-ba0b-225b5c71f798-kube-api-access-8sb85\") pod \"740a1a3b-f68f-438f-ba0b-225b5c71f798\" (UID: \"740a1a3b-f68f-438f-ba0b-225b5c71f798\") " Oct 31 01:26:30.531203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.531148 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-openshift-ca" pod "740a1a3b-f68f-438f-ba0b-225b5c71f798" (UID: "740a1a3b-f68f-438f-ba0b-225b5c71f798"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:30.531401 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.531372 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:26:30.533129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.533101 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-xks-ca" pod "740a1a3b-f68f-438f-ba0b-225b5c71f798" (UID: "740a1a3b-f68f-438f-ba0b-225b5c71f798"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:26:30.533323 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.533156 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740a1a3b-f68f-438f-ba0b-225b5c71f798-kube-api-access-8sb85" pod "740a1a3b-f68f-438f-ba0b-225b5c71f798" (UID: "740a1a3b-f68f-438f-ba0b-225b5c71f798"). InnerVolumeSpecName "kube-api-access-8sb85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:30.632392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.632352 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a1a3b-f68f-438f-ba0b-225b5c71f798-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:26:30.632392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:30.632386 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sb85\" (UniqueName: \"kubernetes.io/projected/740a1a3b-f68f-438f-ba0b-225b5c71f798-kube-api-access-8sb85\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:26:31.236346 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:31.236312 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" event={"ID":"740a1a3b-f68f-438f-ba0b-225b5c71f798","Type":"ContainerDied","Data":"99d4b89aacaf417822ad61503294316c85c63d1abe1b595b68e14cc029170f5d"} Oct 31 01:26:31.236346 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:31.236335 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv" Oct 31 01:26:31.236605 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:31.236363 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d4b89aacaf417822ad61503294316c85c63d1abe1b595b68e14cc029170f5d" Oct 31 01:26:33.181539 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:26:33.181493 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:26:35.817196 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.817155 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q"] Oct 31 01:26:35.817824 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.817802 5978 cpu_manager.go:414] "containerMap: removing container" podUID="740a1a3b-f68f-438f-ba0b-225b5c71f798" containerName="revoke-keys" Oct 31 01:26:35.817931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.817828 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740a1a3b-f68f-438f-ba0b-225b5c71f798" containerName="revoke-keys" Oct 31 01:26:35.817931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.817926 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="740a1a3b-f68f-438f-ba0b-225b5c71f798" containerName="revoke-keys" Oct 31 01:26:35.821269 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.821249 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.848856 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.848824 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q"] Oct 31 01:26:35.878500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.878467 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k785\" (UniqueName: \"kubernetes.io/projected/b6de6a36-1b82-49ef-aa57-c476ced0542f-kube-api-access-8k785\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.878500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.878510 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.878742 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.878554 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.878742 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.878610 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.878816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.878746 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.878816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.878780 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.878816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.878798 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.878940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.878917 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.878988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.878970 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6de6a36-1b82-49ef-aa57-c476ced0542f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.979509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.979422 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.979509 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.979524 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6de6a36-1b82-49ef-aa57-c476ced0542f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.979823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.979562 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k785\" (UniqueName: \"kubernetes.io/projected/b6de6a36-1b82-49ef-aa57-c476ced0542f-kube-api-access-8k785\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.979823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.979583 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.979823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.979606 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.979823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.979622 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.979823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.979703 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.979823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.979733 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.979823 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.979760 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.980200 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.980129 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.980272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.980204 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.980272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.980234 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.980344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.980288 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.980384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.980352 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6de6a36-1b82-49ef-aa57-c476ced0542f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.981962 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.981941 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:35.982258 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:35.982239 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:36.001075 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:36.001040 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6de6a36-1b82-49ef-aa57-c476ced0542f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:36.001189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:36.001115 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k785\" (UniqueName: \"kubernetes.io/projected/b6de6a36-1b82-49ef-aa57-c476ced0542f-kube-api-access-8k785\") pod \"maas-default-gateway-openshift-default-6886b96f9-nqx6q\" (UID: \"b6de6a36-1b82-49ef-aa57-c476ced0542f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:36.135511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:36.135388 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:36.445925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:36.445761 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q"] Oct 31 01:26:36.449140 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:26:36.449111 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6de6a36_1b82_49ef_aa57_c476ced0542f.slice/crio-558e54266a4a1cff52ad2eeb748d6da61aed3485d4b90cc7ad5e63b3e5e652e3 WatchSource:0}: Error finding container 558e54266a4a1cff52ad2eeb748d6da61aed3485d4b90cc7ad5e63b3e5e652e3: Status 404 returned error can't find the container with id 558e54266a4a1cff52ad2eeb748d6da61aed3485d4b90cc7ad5e63b3e5e652e3 Oct 31 01:26:36.451405 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:36.451369 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:26:36.451546 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:36.451439 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:26:36.451546 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:36.451488 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:26:37.273274 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:37.273231 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" event={"ID":"b6de6a36-1b82-49ef-aa57-c476ced0542f","Type":"ContainerStarted","Data":"c6bd2042fa688f740cc634b7b6067984cff78d8222a24aa24613c0eac336a012"} Oct 31 01:26:37.274043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:37.274020 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" event={"ID":"b6de6a36-1b82-49ef-aa57-c476ced0542f","Type":"ContainerStarted","Data":"558e54266a4a1cff52ad2eeb748d6da61aed3485d4b90cc7ad5e63b3e5e652e3"} Oct 31 01:26:37.337478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:37.336997 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podStartSLOduration=2.336969746 podStartE2EDuration="2.336969746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:37.325033378 +0000 UTC m=+4015.023147156" watchObservedRunningTime="2026-10-31 01:26:37.336969746 +0000 UTC m=+4015.035083541" Oct 31 01:26:38.135859 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:38.135818 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:38.137554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:38.137525 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 01:26:38.137691 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:38.137592 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 01:26:39.135988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:39.135938 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 01:26:39.136504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:39.136031 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 01:26:40.137868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:40.137827 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 01:26:40.138493 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:40.138435 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 01:26:41.136976 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:41.136933 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 01:26:41.137807 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:41.137769 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 01:26:42.136178 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:42.136125 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 01:26:42.136742 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:42.136192 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 01:26:43.136545 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:43.136498 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 01:26:43.137112 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:43.136573 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 01:26:45.143846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:45.143801 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:45.144555 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:45.144518 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:46.186676 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:46.186625 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:46.187128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:46.186697 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:47.229412 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:47.228588 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:47.229412 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:47.228654 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:48.269650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:48.269606 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:48.270126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:48.269676 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:49.148884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:49.148837 5978 scope.go:122] "RemoveContainer" containerID="e86aeb54811cadc950e1a26697141c043dbe83ba96c651f08aa449a4d071b06c" Oct 31 01:26:49.264800 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:49.264770 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:49.265084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:49.265065 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:49.265778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:49.265758 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" Oct 31 01:26:50.529154 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.529121 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr"] Oct 31 01:26:50.532951 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.532931 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.545248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.545221 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:26:50.554858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.554828 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:26:50.554858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.554853 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:26:50.555102 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.555078 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:26:50.557505 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.557486 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr"] Oct 31 01:26:50.692975 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.692938 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhvhp\" (UniqueName: \"kubernetes.io/projected/fd502e0d-a72f-4af0-add6-23b84b2da731-kube-api-access-hhvhp\") pod \"maas-api-revoke-sub-2e1f5b5f-bhgqr\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.692975 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.692978 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bhgqr\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.693210 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.693004 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bhgqr\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.794326 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.794248 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhvhp\" (UniqueName: \"kubernetes.io/projected/fd502e0d-a72f-4af0-add6-23b84b2da731-kube-api-access-hhvhp\") pod \"maas-api-revoke-sub-2e1f5b5f-bhgqr\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.794326 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.794284 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bhgqr\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.794326 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.794308 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bhgqr\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.794961 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.794941 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bhgqr\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.796573 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.796555 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bhgqr\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.815130 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.815104 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhvhp\" (UniqueName: \"kubernetes.io/projected/fd502e0d-a72f-4af0-add6-23b84b2da731-kube-api-access-hhvhp\") pod \"maas-api-revoke-sub-2e1f5b5f-bhgqr\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.846526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.846495 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:50.856813 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.856399 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6"] Oct 31 01:26:50.860141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.860124 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:50.884994 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.884960 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6"] Oct 31 01:26:50.997007 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.996970 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:50.997189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.997013 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:50.997189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.997041 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:50.997189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.997154 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppdqq\" (UniqueName: \"kubernetes.io/projected/b4643839-1a91-40bd-b0f2-d335540a1ad5-kube-api-access-ppdqq\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:50.997189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.997186 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:50.997347 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.997216 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:50.997347 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.997250 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:50.997347 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.997304 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:50.997347 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:50.997337 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4643839-1a91-40bd-b0f2-d335540a1ad5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098218 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098260 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098285 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098344 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppdqq\" (UniqueName: \"kubernetes.io/projected/b4643839-1a91-40bd-b0f2-d335540a1ad5-kube-api-access-ppdqq\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098378 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098411 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098434 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098489 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098544 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098541 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4643839-1a91-40bd-b0f2-d335540a1ad5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098706 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.098941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.098934 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.099076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.099054 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.099136 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.099060 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.099352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.099335 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4643839-1a91-40bd-b0f2-d335540a1ad5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.100772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.100752 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.101133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.101115 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.119340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.119311 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4643839-1a91-40bd-b0f2-d335540a1ad5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.119507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.119483 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppdqq\" (UniqueName: \"kubernetes.io/projected/b4643839-1a91-40bd-b0f2-d335540a1ad5-kube-api-access-ppdqq\") pod \"maas-default-gateway-openshift-default-6886b96f9-qq9v6\" (UID: \"b4643839-1a91-40bd-b0f2-d335540a1ad5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.147120 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.147087 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr"] Oct 31 01:26:51.150080 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:26:51.150055 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd502e0d_a72f_4af0_add6_23b84b2da731.slice/crio-595d43b2fb921cdb5eada45a557bdbf22fcec9d248c9fe42f4e4c2e0db58c6b7 WatchSource:0}: Error finding container 595d43b2fb921cdb5eada45a557bdbf22fcec9d248c9fe42f4e4c2e0db58c6b7: Status 404 returned error can't find the container with id 595d43b2fb921cdb5eada45a557bdbf22fcec9d248c9fe42f4e4c2e0db58c6b7 Oct 31 01:26:51.188906 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.188877 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:51.210031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.209998 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:51.392149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.392063 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" event={"ID":"fd502e0d-a72f-4af0-add6-23b84b2da731","Type":"ContainerStarted","Data":"595d43b2fb921cdb5eada45a557bdbf22fcec9d248c9fe42f4e4c2e0db58c6b7"} Oct 31 01:26:51.497782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.497753 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6"] Oct 31 01:26:51.499254 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:26:51.499226 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4643839_1a91_40bd_b0f2_d335540a1ad5.slice/crio-6536174391a180ebbd6e44691e064913108ed54f0b1d84b69dd5b1d5d7cf6fa0 WatchSource:0}: Error finding container 6536174391a180ebbd6e44691e064913108ed54f0b1d84b69dd5b1d5d7cf6fa0: Status 404 returned error can't find the container with id 6536174391a180ebbd6e44691e064913108ed54f0b1d84b69dd5b1d5d7cf6fa0 Oct 31 01:26:51.501556 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.501520 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:26:51.501658 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.501591 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:26:51.501658 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:51.501620 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:26:52.400246 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:52.400143 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" event={"ID":"b4643839-1a91-40bd-b0f2-d335540a1ad5","Type":"ContainerStarted","Data":"c1d3968b14db1449cadd0a394b7d6d1295beeeb0ee95bed48e4bc439386bd231"} Oct 31 01:26:52.400709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:52.400243 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" event={"ID":"b4643839-1a91-40bd-b0f2-d335540a1ad5","Type":"ContainerStarted","Data":"6536174391a180ebbd6e44691e064913108ed54f0b1d84b69dd5b1d5d7cf6fa0"} Oct 31 01:26:52.401827 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:52.401798 5978 generic.go:356] "Generic (PLEG): container finished" podID="fd502e0d-a72f-4af0-add6-23b84b2da731" containerID="4d9831a016fb55c40571ebe7a0b52fe68a01d55937f273ce05424bea5f53bcaa" exitCode=0 Oct 31 01:26:52.401924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:52.401850 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" event={"ID":"fd502e0d-a72f-4af0-add6-23b84b2da731","Type":"ContainerDied","Data":"4d9831a016fb55c40571ebe7a0b52fe68a01d55937f273ce05424bea5f53bcaa"} Oct 31 01:26:52.474496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:52.474414 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" podStartSLOduration=2.474400743 podStartE2EDuration="2.474400743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:52.463208406 +0000 UTC m=+4030.161322172" watchObservedRunningTime="2026-10-31 01:26:52.474400743 +0000 UTC m=+4030.172514507" Oct 31 01:26:53.173404 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.173370 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:53.189615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.189587 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:53.191194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.191168 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qq9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 31 01:26:53.191320 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.191224 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" podUID="b4643839-1a91-40bd-b0f2-d335540a1ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 31 01:26:53.336988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.336952 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:53.591354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.591325 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:53.748859 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.748753 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-openshift-ca\") pod \"fd502e0d-a72f-4af0-add6-23b84b2da731\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " Oct 31 01:26:53.748859 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.748806 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd502e0d-a72f-4af0-add6-23b84b2da731-kube-api-access-hhvhp\" (UniqueName: \"kubernetes.io/projected/fd502e0d-a72f-4af0-add6-23b84b2da731-kube-api-access-hhvhp\") pod \"fd502e0d-a72f-4af0-add6-23b84b2da731\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " Oct 31 01:26:53.749105 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.748945 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-xks-ca\") pod \"fd502e0d-a72f-4af0-add6-23b84b2da731\" (UID: \"fd502e0d-a72f-4af0-add6-23b84b2da731\") " Oct 31 01:26:53.749186 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.749157 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-openshift-ca" pod "fd502e0d-a72f-4af0-add6-23b84b2da731" (UID: "fd502e0d-a72f-4af0-add6-23b84b2da731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:53.749360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.749343 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:26:53.750940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.750918 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd502e0d-a72f-4af0-add6-23b84b2da731-kube-api-access-hhvhp" pod "fd502e0d-a72f-4af0-add6-23b84b2da731" (UID: "fd502e0d-a72f-4af0-add6-23b84b2da731"). InnerVolumeSpecName "kube-api-access-hhvhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:53.751017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.750965 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-xks-ca" pod "fd502e0d-a72f-4af0-add6-23b84b2da731" (UID: "fd502e0d-a72f-4af0-add6-23b84b2da731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:26:53.850003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.849968 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhvhp\" (UniqueName: \"kubernetes.io/projected/fd502e0d-a72f-4af0-add6-23b84b2da731-kube-api-access-hhvhp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:26:53.850003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:53.849999 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd502e0d-a72f-4af0-add6-23b84b2da731-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:26:54.190298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:54.190258 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qq9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 31 01:26:54.190519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:54.190319 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" podUID="b4643839-1a91-40bd-b0f2-d335540a1ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 31 01:26:54.440242 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:54.440211 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" Oct 31 01:26:54.441354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:54.441293 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr" event={"ID":"fd502e0d-a72f-4af0-add6-23b84b2da731","Type":"ContainerDied","Data":"595d43b2fb921cdb5eada45a557bdbf22fcec9d248c9fe42f4e4c2e0db58c6b7"} Oct 31 01:26:54.441354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:54.441337 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="595d43b2fb921cdb5eada45a557bdbf22fcec9d248c9fe42f4e4c2e0db58c6b7" Oct 31 01:26:55.190303 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:55.190265 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qq9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 31 01:26:55.190764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:55.190348 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" podUID="b4643839-1a91-40bd-b0f2-d335540a1ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 31 01:26:56.190352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:56.190305 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qq9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 31 01:26:56.190903 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:56.190385 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" podUID="b4643839-1a91-40bd-b0f2-d335540a1ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 31 01:26:57.189370 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:57.189328 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qq9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 31 01:26:57.189563 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:57.189392 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" podUID="b4643839-1a91-40bd-b0f2-d335540a1ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 31 01:26:58.189965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:58.189920 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qq9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 31 01:26:58.190506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:58.189995 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" podUID="b4643839-1a91-40bd-b0f2-d335540a1ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 31 01:26:59.746429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:59.746395 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:59.746979 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:59.746823 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:26:59.747613 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:26:59.747590 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" Oct 31 01:27:01.827269 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.827198 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4"] Oct 31 01:27:01.829890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.828108 5978 cpu_manager.go:414] "containerMap: removing container" podUID="fd502e0d-a72f-4af0-add6-23b84b2da731" containerName="revoke-keys" Oct 31 01:27:01.829890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.828130 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd502e0d-a72f-4af0-add6-23b84b2da731" containerName="revoke-keys" Oct 31 01:27:01.829890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.828238 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd502e0d-a72f-4af0-add6-23b84b2da731" containerName="revoke-keys" Oct 31 01:27:01.831355 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.831337 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:01.854962 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.854926 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:27:01.854962 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.854934 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:27:01.855223 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.855072 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:27:01.855312 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.855292 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:27:01.857154 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.857128 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4"] Oct 31 01:27:01.929300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.929258 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-29rs4\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:01.929501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.929309 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-29rs4\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:01.929501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:01.929401 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqjjj\" (UniqueName: \"kubernetes.io/projected/47e18e8c-e0b4-48ef-b949-cf11571b7e30-kube-api-access-kqjjj\") pod \"maas-api-revoke-sub-ec843018-29rs4\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:02.030168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:02.030124 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-29rs4\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:02.030168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:02.030163 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-29rs4\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:02.030416 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:02.030188 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqjjj\" (UniqueName: \"kubernetes.io/projected/47e18e8c-e0b4-48ef-b949-cf11571b7e30-kube-api-access-kqjjj\") pod \"maas-api-revoke-sub-ec843018-29rs4\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:02.030839 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:02.030819 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-29rs4\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:02.032694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:02.032673 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-29rs4\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:02.052111 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:02.052067 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqjjj\" (UniqueName: \"kubernetes.io/projected/47e18e8c-e0b4-48ef-b949-cf11571b7e30-kube-api-access-kqjjj\") pod \"maas-api-revoke-sub-ec843018-29rs4\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:02.145658 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:02.145568 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:02.446309 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:02.446280 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4"] Oct 31 01:27:02.448872 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:27:02.448841 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e18e8c_e0b4_48ef_b949_cf11571b7e30.slice/crio-d55c71807acaa1c0dc44111fe7cba3abe914c57127fa22f8929d271ba821abbc WatchSource:0}: Error finding container d55c71807acaa1c0dc44111fe7cba3abe914c57127fa22f8929d271ba821abbc: Status 404 returned error can't find the container with id d55c71807acaa1c0dc44111fe7cba3abe914c57127fa22f8929d271ba821abbc Oct 31 01:27:02.495639 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:02.495600 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" event={"ID":"47e18e8c-e0b4-48ef-b949-cf11571b7e30","Type":"ContainerStarted","Data":"d55c71807acaa1c0dc44111fe7cba3abe914c57127fa22f8929d271ba821abbc"} Oct 31 01:27:03.212389 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:27:03.212352 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:27:03.502606 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:03.502508 5978 generic.go:356] "Generic (PLEG): container finished" podID="47e18e8c-e0b4-48ef-b949-cf11571b7e30" containerID="bf7f62cf77d6b02bb754ef7c482664366d36f6f78d6b65a1409d2461d09868d6" exitCode=0 Oct 31 01:27:03.502774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:03.502594 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" event={"ID":"47e18e8c-e0b4-48ef-b949-cf11571b7e30","Type":"ContainerDied","Data":"bf7f62cf77d6b02bb754ef7c482664366d36f6f78d6b65a1409d2461d09868d6"} Oct 31 01:27:04.643701 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.643672 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:04.661194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.661164 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e18e8c-e0b4-48ef-b949-cf11571b7e30-kube-api-access-kqjjj\" (UniqueName: \"kubernetes.io/projected/47e18e8c-e0b4-48ef-b949-cf11571b7e30-kube-api-access-kqjjj\") pod \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " Oct 31 01:27:04.661307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.661218 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-xks-ca\") pod \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " Oct 31 01:27:04.661307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.661292 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-openshift-ca\") pod \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\" (UID: \"47e18e8c-e0b4-48ef-b949-cf11571b7e30\") " Oct 31 01:27:04.661694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.661665 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-openshift-ca" pod "47e18e8c-e0b4-48ef-b949-cf11571b7e30" (UID: "47e18e8c-e0b4-48ef-b949-cf11571b7e30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:04.661792 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.661748 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:27:04.663363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.663340 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-xks-ca" pod "47e18e8c-e0b4-48ef-b949-cf11571b7e30" (UID: "47e18e8c-e0b4-48ef-b949-cf11571b7e30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:04.663480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.663417 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e18e8c-e0b4-48ef-b949-cf11571b7e30-kube-api-access-kqjjj" pod "47e18e8c-e0b4-48ef-b949-cf11571b7e30" (UID: "47e18e8c-e0b4-48ef-b949-cf11571b7e30"). InnerVolumeSpecName "kube-api-access-kqjjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:04.763344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.763258 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e18e8c-e0b4-48ef-b949-cf11571b7e30-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:27:04.763344 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:04.763291 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqjjj\" (UniqueName: \"kubernetes.io/projected/47e18e8c-e0b4-48ef-b949-cf11571b7e30-kube-api-access-kqjjj\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:27:05.514347 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:05.514316 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" Oct 31 01:27:05.514605 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:05.514355 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4" event={"ID":"47e18e8c-e0b4-48ef-b949-cf11571b7e30","Type":"ContainerDied","Data":"d55c71807acaa1c0dc44111fe7cba3abe914c57127fa22f8929d271ba821abbc"} Oct 31 01:27:05.514605 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:05.514401 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d55c71807acaa1c0dc44111fe7cba3abe914c57127fa22f8929d271ba821abbc" Oct 31 01:27:20.329494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:20.326565 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nqx6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:27:20.329494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:20.326653 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nqx6q" podUID="b6de6a36-1b82-49ef-aa57-c476ced0542f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:27:29.916227 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.915804 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq"] Oct 31 01:27:29.917218 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.917196 5978 cpu_manager.go:414] "containerMap: removing container" podUID="47e18e8c-e0b4-48ef-b949-cf11571b7e30" containerName="revoke-keys" Oct 31 01:27:29.917218 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.917219 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e18e8c-e0b4-48ef-b949-cf11571b7e30" containerName="revoke-keys" Oct 31 01:27:29.917339 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.917306 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e18e8c-e0b4-48ef-b949-cf11571b7e30" containerName="revoke-keys" Oct 31 01:27:29.920699 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.920680 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:29.932997 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.932974 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:27:29.933121 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.933036 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:27:29.942725 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.942689 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:27:29.942867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.942701 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:27:29.944884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:29.944859 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq"] Oct 31 01:27:30.072153 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.072123 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-25skq\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.072361 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.072164 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-25skq\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.072361 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.072186 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tr2g\" (UniqueName: \"kubernetes.io/projected/77be76af-a574-49c2-8979-32e92e31c33a-kube-api-access-4tr2g\") pod \"maas-api-revoke-sub-046e1343-25skq\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.173522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.173406 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-25skq\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.173522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.173490 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-25skq\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.173522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.173522 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tr2g\" (UniqueName: \"kubernetes.io/projected/77be76af-a574-49c2-8979-32e92e31c33a-kube-api-access-4tr2g\") pod \"maas-api-revoke-sub-046e1343-25skq\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.174126 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.174102 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-25skq\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.175903 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.175881 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-25skq\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.194854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.194819 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tr2g\" (UniqueName: \"kubernetes.io/projected/77be76af-a574-49c2-8979-32e92e31c33a-kube-api-access-4tr2g\") pod \"maas-api-revoke-sub-046e1343-25skq\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.233534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.233502 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:30.528106 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.528071 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq"] Oct 31 01:27:30.530778 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:27:30.530748 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77be76af_a574_49c2_8979_32e92e31c33a.slice/crio-10cabbe49c2f8a8b0d5ac0ab36fbb55db338fb7e1b81ffdf9cea85ad62de0ca7 WatchSource:0}: Error finding container 10cabbe49c2f8a8b0d5ac0ab36fbb55db338fb7e1b81ffdf9cea85ad62de0ca7: Status 404 returned error can't find the container with id 10cabbe49c2f8a8b0d5ac0ab36fbb55db338fb7e1b81ffdf9cea85ad62de0ca7 Oct 31 01:27:30.677029 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:30.676994 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" event={"ID":"77be76af-a574-49c2-8979-32e92e31c33a","Type":"ContainerStarted","Data":"10cabbe49c2f8a8b0d5ac0ab36fbb55db338fb7e1b81ffdf9cea85ad62de0ca7"} Oct 31 01:27:31.683519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:31.683414 5978 generic.go:356] "Generic (PLEG): container finished" podID="77be76af-a574-49c2-8979-32e92e31c33a" containerID="a8ce3cf5a028dc4e648d940b0072ae0729d6c0298827a4b6388f857967060dcb" exitCode=0 Oct 31 01:27:31.683894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:31.683507 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" event={"ID":"77be76af-a574-49c2-8979-32e92e31c33a","Type":"ContainerDied","Data":"a8ce3cf5a028dc4e648d940b0072ae0729d6c0298827a4b6388f857967060dcb"} Oct 31 01:27:32.822412 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:32.822387 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:32.903342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:32.903308 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-xks-ca\") pod \"77be76af-a574-49c2-8979-32e92e31c33a\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " Oct 31 01:27:32.903533 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:32.903385 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77be76af-a574-49c2-8979-32e92e31c33a-kube-api-access-4tr2g\" (UniqueName: \"kubernetes.io/projected/77be76af-a574-49c2-8979-32e92e31c33a-kube-api-access-4tr2g\") pod \"77be76af-a574-49c2-8979-32e92e31c33a\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " Oct 31 01:27:32.903533 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:32.903522 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-openshift-ca\") pod \"77be76af-a574-49c2-8979-32e92e31c33a\" (UID: \"77be76af-a574-49c2-8979-32e92e31c33a\") " Oct 31 01:27:32.903895 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:32.903865 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-openshift-ca" pod "77be76af-a574-49c2-8979-32e92e31c33a" (UID: "77be76af-a574-49c2-8979-32e92e31c33a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:32.905624 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:32.905596 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-xks-ca" pod "77be76af-a574-49c2-8979-32e92e31c33a" (UID: "77be76af-a574-49c2-8979-32e92e31c33a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:32.905724 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:32.905667 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77be76af-a574-49c2-8979-32e92e31c33a-kube-api-access-4tr2g" pod "77be76af-a574-49c2-8979-32e92e31c33a" (UID: "77be76af-a574-49c2-8979-32e92e31c33a"). InnerVolumeSpecName "kube-api-access-4tr2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:33.004489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:33.004381 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:27:33.004489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:33.004412 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tr2g\" (UniqueName: \"kubernetes.io/projected/77be76af-a574-49c2-8979-32e92e31c33a-kube-api-access-4tr2g\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:27:33.004489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:33.004422 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77be76af-a574-49c2-8979-32e92e31c33a-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:27:33.180865 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:27:33.180825 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:27:33.700842 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:33.700801 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" event={"ID":"77be76af-a574-49c2-8979-32e92e31c33a","Type":"ContainerDied","Data":"10cabbe49c2f8a8b0d5ac0ab36fbb55db338fb7e1b81ffdf9cea85ad62de0ca7"} Oct 31 01:27:33.701042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:33.700866 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq" Oct 31 01:27:33.701042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:33.700884 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10cabbe49c2f8a8b0d5ac0ab36fbb55db338fb7e1b81ffdf9cea85ad62de0ca7" Oct 31 01:27:40.267983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.267937 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb"] Oct 31 01:27:40.268557 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.268539 5978 cpu_manager.go:414] "containerMap: removing container" podUID="77be76af-a574-49c2-8979-32e92e31c33a" containerName="revoke-keys" Oct 31 01:27:40.268626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.268560 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77be76af-a574-49c2-8979-32e92e31c33a" containerName="revoke-keys" Oct 31 01:27:40.268713 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.268657 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="77be76af-a574-49c2-8979-32e92e31c33a" containerName="revoke-keys" Oct 31 01:27:40.272187 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.272170 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.278221 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.278192 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2j8n\" (UniqueName: \"kubernetes.io/projected/20a2878f-0abf-4098-b8a1-6ef92cd23591-kube-api-access-t2j8n\") pod \"maas-api-revoke-sub-d3fba056-r45wb\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.278340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.278254 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-r45wb\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.278340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.278327 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-r45wb\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.284640 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.284617 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:27:40.284750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.284656 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:27:40.294537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.294180 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:27:40.294537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.294292 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:27:40.296814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.296792 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb"] Oct 31 01:27:40.379545 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.379513 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2j8n\" (UniqueName: \"kubernetes.io/projected/20a2878f-0abf-4098-b8a1-6ef92cd23591-kube-api-access-t2j8n\") pod \"maas-api-revoke-sub-d3fba056-r45wb\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.379760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.379560 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-r45wb\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.379760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.379584 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-r45wb\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.380188 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.380167 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-r45wb\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.381891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.381872 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-r45wb\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.400499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.400469 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2j8n\" (UniqueName: \"kubernetes.io/projected/20a2878f-0abf-4098-b8a1-6ef92cd23591-kube-api-access-t2j8n\") pod \"maas-api-revoke-sub-d3fba056-r45wb\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.586031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.585994 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:40.883536 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:40.883512 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb"] Oct 31 01:27:40.886105 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:27:40.886078 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a2878f_0abf_4098_b8a1_6ef92cd23591.slice/crio-98c0c3d2d259f194413f2bba1808053d0235bd6243b2f72a92c50f28987143ed WatchSource:0}: Error finding container 98c0c3d2d259f194413f2bba1808053d0235bd6243b2f72a92c50f28987143ed: Status 404 returned error can't find the container with id 98c0c3d2d259f194413f2bba1808053d0235bd6243b2f72a92c50f28987143ed Oct 31 01:27:41.748388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:41.748333 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" event={"ID":"20a2878f-0abf-4098-b8a1-6ef92cd23591","Type":"ContainerStarted","Data":"ddd715c2420a5f7314da6a8b752a651081a0fa7d20c0f5c4192aee1d3623f91a"} Oct 31 01:27:41.748975 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:41.748420 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" event={"ID":"20a2878f-0abf-4098-b8a1-6ef92cd23591","Type":"ContainerStarted","Data":"98c0c3d2d259f194413f2bba1808053d0235bd6243b2f72a92c50f28987143ed"} Oct 31 01:27:41.784521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:41.784434 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" podStartSLOduration=1.784415243 podStartE2EDuration="1.784415243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:41.783384845 +0000 UTC m=+4079.481498611" watchObservedRunningTime="2026-10-31 01:27:41.784415243 +0000 UTC m=+4079.482529008" Oct 31 01:27:42.775052 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:42.775012 5978 generic.go:356] "Generic (PLEG): container finished" podID="20a2878f-0abf-4098-b8a1-6ef92cd23591" containerID="ddd715c2420a5f7314da6a8b752a651081a0fa7d20c0f5c4192aee1d3623f91a" exitCode=0 Oct 31 01:27:42.775670 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:42.775644 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" event={"ID":"20a2878f-0abf-4098-b8a1-6ef92cd23591","Type":"ContainerDied","Data":"ddd715c2420a5f7314da6a8b752a651081a0fa7d20c0f5c4192aee1d3623f91a"} Oct 31 01:27:44.460844 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.460813 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:44.596008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.595969 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-openshift-ca\") pod \"20a2878f-0abf-4098-b8a1-6ef92cd23591\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " Oct 31 01:27:44.596215 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.596050 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-xks-ca\") pod \"20a2878f-0abf-4098-b8a1-6ef92cd23591\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " Oct 31 01:27:44.596215 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.596151 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a2878f-0abf-4098-b8a1-6ef92cd23591-kube-api-access-t2j8n\" (UniqueName: \"kubernetes.io/projected/20a2878f-0abf-4098-b8a1-6ef92cd23591-kube-api-access-t2j8n\") pod \"20a2878f-0abf-4098-b8a1-6ef92cd23591\" (UID: \"20a2878f-0abf-4098-b8a1-6ef92cd23591\") " Oct 31 01:27:44.596421 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.596383 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-openshift-ca" pod "20a2878f-0abf-4098-b8a1-6ef92cd23591" (UID: "20a2878f-0abf-4098-b8a1-6ef92cd23591"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:44.598146 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.598123 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-xks-ca" pod "20a2878f-0abf-4098-b8a1-6ef92cd23591" (UID: "20a2878f-0abf-4098-b8a1-6ef92cd23591"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:44.598250 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.598236 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a2878f-0abf-4098-b8a1-6ef92cd23591-kube-api-access-t2j8n" pod "20a2878f-0abf-4098-b8a1-6ef92cd23591" (UID: "20a2878f-0abf-4098-b8a1-6ef92cd23591"). InnerVolumeSpecName "kube-api-access-t2j8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:44.697782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.697713 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2j8n\" (UniqueName: \"kubernetes.io/projected/20a2878f-0abf-4098-b8a1-6ef92cd23591-kube-api-access-t2j8n\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:27:44.697782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.697739 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:27:44.697782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.697749 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a2878f-0abf-4098-b8a1-6ef92cd23591-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:27:44.813655 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.813628 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" Oct 31 01:27:44.813655 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.813645 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb" event={"ID":"20a2878f-0abf-4098-b8a1-6ef92cd23591","Type":"ContainerDied","Data":"98c0c3d2d259f194413f2bba1808053d0235bd6243b2f72a92c50f28987143ed"} Oct 31 01:27:44.813877 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:44.813691 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98c0c3d2d259f194413f2bba1808053d0235bd6243b2f72a92c50f28987143ed" Oct 31 01:27:51.335899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:27:51.335864 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:02.314963 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.314927 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6"] Oct 31 01:28:02.315629 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.315611 5978 cpu_manager.go:414] "containerMap: removing container" podUID="20a2878f-0abf-4098-b8a1-6ef92cd23591" containerName="revoke-keys" Oct 31 01:28:02.315684 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.315632 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a2878f-0abf-4098-b8a1-6ef92cd23591" containerName="revoke-keys" Oct 31 01:28:02.315741 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.315731 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a2878f-0abf-4098-b8a1-6ef92cd23591" containerName="revoke-keys" Oct 31 01:28:02.318740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.318725 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.331116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.331092 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:28:02.334493 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.334447 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lljj6\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.334603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.334535 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l7j6\" (UniqueName: \"kubernetes.io/projected/8224548f-686b-41cd-b027-98b0daa85c2f-kube-api-access-6l7j6\") pod \"maas-api-revoke-sub-ae594979-lljj6\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.334603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.334583 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lljj6\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.340595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.340545 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:28:02.340595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.340557 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:28:02.340750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.340682 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:28:02.343519 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.343495 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6"] Oct 31 01:28:02.435708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.435665 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lljj6\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.435925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.435813 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lljj6\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.435925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.435850 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l7j6\" (UniqueName: \"kubernetes.io/projected/8224548f-686b-41cd-b027-98b0daa85c2f-kube-api-access-6l7j6\") pod \"maas-api-revoke-sub-ae594979-lljj6\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.436356 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.436331 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lljj6\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.438198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.438176 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lljj6\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.457043 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.457011 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l7j6\" (UniqueName: \"kubernetes.io/projected/8224548f-686b-41cd-b027-98b0daa85c2f-kube-api-access-6l7j6\") pod \"maas-api-revoke-sub-ae594979-lljj6\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.636431 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.636394 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:02.930063 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.930014 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6"] Oct 31 01:28:02.932519 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:28:02.932491 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8224548f_686b_41cd_b027_98b0daa85c2f.slice/crio-54269eedf07feb2db1f8fd29c2a9b26092cbda13ea8247289942ef1c794d5b3a WatchSource:0}: Error finding container 54269eedf07feb2db1f8fd29c2a9b26092cbda13ea8247289942ef1c794d5b3a: Status 404 returned error can't find the container with id 54269eedf07feb2db1f8fd29c2a9b26092cbda13ea8247289942ef1c794d5b3a Oct 31 01:28:02.960996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:02.960968 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" event={"ID":"8224548f-686b-41cd-b027-98b0daa85c2f","Type":"ContainerStarted","Data":"54269eedf07feb2db1f8fd29c2a9b26092cbda13ea8247289942ef1c794d5b3a"} Oct 31 01:28:03.233143 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:28:03.233102 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:28:03.967774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:03.967691 5978 generic.go:356] "Generic (PLEG): container finished" podID="8224548f-686b-41cd-b027-98b0daa85c2f" containerID="25c6b90973bf35a6840da05ae0ea4bcbaedcffd02b1c83674d2bbb048a7b4403" exitCode=0 Oct 31 01:28:03.967774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:03.967760 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" event={"ID":"8224548f-686b-41cd-b027-98b0daa85c2f","Type":"ContainerDied","Data":"25c6b90973bf35a6840da05ae0ea4bcbaedcffd02b1c83674d2bbb048a7b4403"} Oct 31 01:28:05.108014 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.107990 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:05.167411 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.167381 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-xks-ca\") pod \"8224548f-686b-41cd-b027-98b0daa85c2f\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " Oct 31 01:28:05.167600 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.167497 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-openshift-ca\") pod \"8224548f-686b-41cd-b027-98b0daa85c2f\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " Oct 31 01:28:05.167600 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.167539 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8224548f-686b-41cd-b027-98b0daa85c2f-kube-api-access-6l7j6\" (UniqueName: \"kubernetes.io/projected/8224548f-686b-41cd-b027-98b0daa85c2f-kube-api-access-6l7j6\") pod \"8224548f-686b-41cd-b027-98b0daa85c2f\" (UID: \"8224548f-686b-41cd-b027-98b0daa85c2f\") " Oct 31 01:28:05.167884 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.167852 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-openshift-ca" pod "8224548f-686b-41cd-b027-98b0daa85c2f" (UID: "8224548f-686b-41cd-b027-98b0daa85c2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:05.169697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.169668 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-xks-ca" pod "8224548f-686b-41cd-b027-98b0daa85c2f" (UID: "8224548f-686b-41cd-b027-98b0daa85c2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:05.169697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.169669 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8224548f-686b-41cd-b027-98b0daa85c2f-kube-api-access-6l7j6" pod "8224548f-686b-41cd-b027-98b0daa85c2f" (UID: "8224548f-686b-41cd-b027-98b0daa85c2f"). InnerVolumeSpecName "kube-api-access-6l7j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:05.269562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.269440 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:28:05.269562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.269512 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8224548f-686b-41cd-b027-98b0daa85c2f-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:28:05.269562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.269524 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l7j6\" (UniqueName: \"kubernetes.io/projected/8224548f-686b-41cd-b027-98b0daa85c2f-kube-api-access-6l7j6\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:28:05.979683 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.979648 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" Oct 31 01:28:05.979889 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.979649 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6" event={"ID":"8224548f-686b-41cd-b027-98b0daa85c2f","Type":"ContainerDied","Data":"54269eedf07feb2db1f8fd29c2a9b26092cbda13ea8247289942ef1c794d5b3a"} Oct 31 01:28:05.979889 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:05.979828 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54269eedf07feb2db1f8fd29c2a9b26092cbda13ea8247289942ef1c794d5b3a" Oct 31 01:28:24.011149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.011113 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95"] Oct 31 01:28:24.012314 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.012287 5978 cpu_manager.go:414] "containerMap: removing container" podUID="8224548f-686b-41cd-b027-98b0daa85c2f" containerName="revoke-keys" Oct 31 01:28:24.012314 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.012313 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8224548f-686b-41cd-b027-98b0daa85c2f" containerName="revoke-keys" Oct 31 01:28:24.012510 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.012411 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="8224548f-686b-41cd-b027-98b0daa85c2f" containerName="revoke-keys" Oct 31 01:28:24.015580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.015556 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.028002 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.027978 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:28:24.028135 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.027996 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:28:24.037729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.037686 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:28:24.037909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.037735 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:28:24.041138 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.041106 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95"] Oct 31 01:28:24.117363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.117325 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ghzw\" (UniqueName: \"kubernetes.io/projected/950786e0-b57f-4452-8028-2cdd94b6cad7-kube-api-access-4ghzw\") pod \"maas-api-revoke-sub-a4aef7d7-q4k95\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.117583 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.117382 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q4k95\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.117583 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.117499 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q4k95\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.218959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.218925 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ghzw\" (UniqueName: \"kubernetes.io/projected/950786e0-b57f-4452-8028-2cdd94b6cad7-kube-api-access-4ghzw\") pod \"maas-api-revoke-sub-a4aef7d7-q4k95\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.219181 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.218984 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q4k95\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.219181 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.219040 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q4k95\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.219685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.219660 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q4k95\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.221408 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.221385 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-q4k95\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.240476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.240432 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ghzw\" (UniqueName: \"kubernetes.io/projected/950786e0-b57f-4452-8028-2cdd94b6cad7-kube-api-access-4ghzw\") pod \"maas-api-revoke-sub-a4aef7d7-q4k95\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.329531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.329443 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:24.624946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:24.624753 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95"] Oct 31 01:28:24.627930 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:28:24.627900 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950786e0_b57f_4452_8028_2cdd94b6cad7.slice/crio-c1bd5a883fbf93c5d84677cc698be882a1e38a935594cf49a51c39ffac5a6a7c WatchSource:0}: Error finding container c1bd5a883fbf93c5d84677cc698be882a1e38a935594cf49a51c39ffac5a6a7c: Status 404 returned error can't find the container with id c1bd5a883fbf93c5d84677cc698be882a1e38a935594cf49a51c39ffac5a6a7c Oct 31 01:28:25.139866 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:25.139826 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" event={"ID":"950786e0-b57f-4452-8028-2cdd94b6cad7","Type":"ContainerStarted","Data":"71a333ca7a8441954dde90156d01d179ae0354a6926588d013d8fd19045ad3de"} Oct 31 01:28:25.140313 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:25.139884 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" event={"ID":"950786e0-b57f-4452-8028-2cdd94b6cad7","Type":"ContainerStarted","Data":"c1bd5a883fbf93c5d84677cc698be882a1e38a935594cf49a51c39ffac5a6a7c"} Oct 31 01:28:25.177365 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:25.177246 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" podStartSLOduration=2.177228407 podStartE2EDuration="2.177228407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:25.175064841 +0000 UTC m=+4122.873178605" watchObservedRunningTime="2026-10-31 01:28:25.177228407 +0000 UTC m=+4122.875342172" Oct 31 01:28:26.146854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:26.146809 5978 generic.go:356] "Generic (PLEG): container finished" podID="950786e0-b57f-4452-8028-2cdd94b6cad7" containerID="71a333ca7a8441954dde90156d01d179ae0354a6926588d013d8fd19045ad3de" exitCode=0 Oct 31 01:28:26.147294 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:26.146902 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" event={"ID":"950786e0-b57f-4452-8028-2cdd94b6cad7","Type":"ContainerDied","Data":"71a333ca7a8441954dde90156d01d179ae0354a6926588d013d8fd19045ad3de"} Oct 31 01:28:27.296927 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.296897 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:27.460634 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.460541 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-openshift-ca\") pod \"950786e0-b57f-4452-8028-2cdd94b6cad7\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " Oct 31 01:28:27.460829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.460664 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-xks-ca\") pod \"950786e0-b57f-4452-8028-2cdd94b6cad7\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " Oct 31 01:28:27.460829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.460777 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950786e0-b57f-4452-8028-2cdd94b6cad7-kube-api-access-4ghzw\" (UniqueName: \"kubernetes.io/projected/950786e0-b57f-4452-8028-2cdd94b6cad7-kube-api-access-4ghzw\") pod \"950786e0-b57f-4452-8028-2cdd94b6cad7\" (UID: \"950786e0-b57f-4452-8028-2cdd94b6cad7\") " Oct 31 01:28:27.460993 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.460970 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-openshift-ca" pod "950786e0-b57f-4452-8028-2cdd94b6cad7" (UID: "950786e0-b57f-4452-8028-2cdd94b6cad7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:27.461157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.461141 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:28:27.462818 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.462782 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-xks-ca" pod "950786e0-b57f-4452-8028-2cdd94b6cad7" (UID: "950786e0-b57f-4452-8028-2cdd94b6cad7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:27.462938 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.462916 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950786e0-b57f-4452-8028-2cdd94b6cad7-kube-api-access-4ghzw" pod "950786e0-b57f-4452-8028-2cdd94b6cad7" (UID: "950786e0-b57f-4452-8028-2cdd94b6cad7"). InnerVolumeSpecName "kube-api-access-4ghzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:27.562738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.562683 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/950786e0-b57f-4452-8028-2cdd94b6cad7-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:28:27.562738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:27.562733 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ghzw\" (UniqueName: \"kubernetes.io/projected/950786e0-b57f-4452-8028-2cdd94b6cad7-kube-api-access-4ghzw\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:28:28.158660 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:28.158627 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" Oct 31 01:28:28.158865 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:28.158656 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95" event={"ID":"950786e0-b57f-4452-8028-2cdd94b6cad7","Type":"ContainerDied","Data":"c1bd5a883fbf93c5d84677cc698be882a1e38a935594cf49a51c39ffac5a6a7c"} Oct 31 01:28:28.158865 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:28.158716 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1bd5a883fbf93c5d84677cc698be882a1e38a935594cf49a51c39ffac5a6a7c" Oct 31 01:28:31.032739 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:31.032705 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65"] Oct 31 01:28:31.042755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:31.042721 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc4b638-26s65"] Oct 31 01:28:32.423692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:32.423644 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5e32a8-8cec-42f8-8264-35956dca8e0c" path="/var/lib/kubelet/pods/0d5e32a8-8cec-42f8-8264-35956dca8e0c/volumes" Oct 31 01:28:33.181219 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:28:33.181174 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:28:45.284360 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.284322 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9"] Oct 31 01:28:45.285034 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.285013 5978 cpu_manager.go:414] "containerMap: removing container" podUID="950786e0-b57f-4452-8028-2cdd94b6cad7" containerName="revoke-keys" Oct 31 01:28:45.285120 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.285038 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950786e0-b57f-4452-8028-2cdd94b6cad7" containerName="revoke-keys" Oct 31 01:28:45.285209 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.285145 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="950786e0-b57f-4452-8028-2cdd94b6cad7" containerName="revoke-keys" Oct 31 01:28:45.289769 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.289746 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.302236 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.302213 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:28:45.302346 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.302250 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:28:45.312309 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.312278 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:28:45.312490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.312389 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:28:45.314804 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.314778 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9"] Oct 31 01:28:45.421835 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.421799 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbp7c\" (UniqueName: \"kubernetes.io/projected/329d88be-8e2c-4b9c-b74b-474f12957f5a-kube-api-access-rbp7c\") pod \"maas-api-revoke-sub-79805ca9-chxk9\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.422033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.421972 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-chxk9\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.422033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.422027 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-chxk9\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.523150 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.523111 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-chxk9\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.523321 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.523169 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-chxk9\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.523321 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.523215 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbp7c\" (UniqueName: \"kubernetes.io/projected/329d88be-8e2c-4b9c-b74b-474f12957f5a-kube-api-access-rbp7c\") pod \"maas-api-revoke-sub-79805ca9-chxk9\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.523906 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.523883 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-chxk9\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.525524 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.525504 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-chxk9\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.544434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.544371 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbp7c\" (UniqueName: \"kubernetes.io/projected/329d88be-8e2c-4b9c-b74b-474f12957f5a-kube-api-access-rbp7c\") pod \"maas-api-revoke-sub-79805ca9-chxk9\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.603225 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.603186 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:45.898841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:45.898580 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9"] Oct 31 01:28:45.900840 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:28:45.900805 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329d88be_8e2c_4b9c_b74b_474f12957f5a.slice/crio-35c2dcc6085e8afb5277dd817f95c2b244152bf637173ba37d7b511d06c46def WatchSource:0}: Error finding container 35c2dcc6085e8afb5277dd817f95c2b244152bf637173ba37d7b511d06c46def: Status 404 returned error can't find the container with id 35c2dcc6085e8afb5277dd817f95c2b244152bf637173ba37d7b511d06c46def Oct 31 01:28:46.316895 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:46.316854 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" event={"ID":"329d88be-8e2c-4b9c-b74b-474f12957f5a","Type":"ContainerStarted","Data":"0d883e1f30132074d256b6a6134682c7678fe613481578ba3d30b6d8fa78f20f"} Oct 31 01:28:46.317347 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:46.316923 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" event={"ID":"329d88be-8e2c-4b9c-b74b-474f12957f5a","Type":"ContainerStarted","Data":"35c2dcc6085e8afb5277dd817f95c2b244152bf637173ba37d7b511d06c46def"} Oct 31 01:28:46.354335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:46.354278 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" podStartSLOduration=1.3542645150000001 podStartE2EDuration="1.354264515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:46.352489432 +0000 UTC m=+4144.050603198" watchObservedRunningTime="2026-10-31 01:28:46.354264515 +0000 UTC m=+4144.052378279" Oct 31 01:28:47.324711 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:47.324672 5978 generic.go:356] "Generic (PLEG): container finished" podID="329d88be-8e2c-4b9c-b74b-474f12957f5a" containerID="0d883e1f30132074d256b6a6134682c7678fe613481578ba3d30b6d8fa78f20f" exitCode=0 Oct 31 01:28:47.325130 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:47.324753 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" event={"ID":"329d88be-8e2c-4b9c-b74b-474f12957f5a","Type":"ContainerDied","Data":"0d883e1f30132074d256b6a6134682c7678fe613481578ba3d30b6d8fa78f20f"} Oct 31 01:28:48.473175 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.473147 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:48.656066 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.655980 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-xks-ca\") pod \"329d88be-8e2c-4b9c-b74b-474f12957f5a\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " Oct 31 01:28:48.656234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.656176 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-openshift-ca\") pod \"329d88be-8e2c-4b9c-b74b-474f12957f5a\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " Oct 31 01:28:48.656234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.656226 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329d88be-8e2c-4b9c-b74b-474f12957f5a-kube-api-access-rbp7c\" (UniqueName: \"kubernetes.io/projected/329d88be-8e2c-4b9c-b74b-474f12957f5a-kube-api-access-rbp7c\") pod \"329d88be-8e2c-4b9c-b74b-474f12957f5a\" (UID: \"329d88be-8e2c-4b9c-b74b-474f12957f5a\") " Oct 31 01:28:48.656580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.656546 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-openshift-ca" pod "329d88be-8e2c-4b9c-b74b-474f12957f5a" (UID: "329d88be-8e2c-4b9c-b74b-474f12957f5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:48.658131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.658108 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-xks-ca" pod "329d88be-8e2c-4b9c-b74b-474f12957f5a" (UID: "329d88be-8e2c-4b9c-b74b-474f12957f5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:48.658241 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.658225 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329d88be-8e2c-4b9c-b74b-474f12957f5a-kube-api-access-rbp7c" pod "329d88be-8e2c-4b9c-b74b-474f12957f5a" (UID: "329d88be-8e2c-4b9c-b74b-474f12957f5a"). InnerVolumeSpecName "kube-api-access-rbp7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:48.757667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.757630 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:28:48.757667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.757663 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbp7c\" (UniqueName: \"kubernetes.io/projected/329d88be-8e2c-4b9c-b74b-474f12957f5a-kube-api-access-rbp7c\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:28:48.757667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:48.757675 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329d88be-8e2c-4b9c-b74b-474f12957f5a-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:28:49.337998 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:49.337961 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" event={"ID":"329d88be-8e2c-4b9c-b74b-474f12957f5a","Type":"ContainerDied","Data":"35c2dcc6085e8afb5277dd817f95c2b244152bf637173ba37d7b511d06c46def"} Oct 31 01:28:49.338158 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:49.338010 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35c2dcc6085e8afb5277dd817f95c2b244152bf637173ba37d7b511d06c46def" Oct 31 01:28:49.338158 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:49.337978 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9" Oct 31 01:28:51.458902 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:28:51.458869 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:03.181502 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:29:03.181400 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:29:05.349032 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.348995 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn"] Oct 31 01:29:05.349965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.349941 5978 cpu_manager.go:414] "containerMap: removing container" podUID="329d88be-8e2c-4b9c-b74b-474f12957f5a" containerName="revoke-keys" Oct 31 01:29:05.349965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.349965 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329d88be-8e2c-4b9c-b74b-474f12957f5a" containerName="revoke-keys" Oct 31 01:29:05.350124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.350066 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="329d88be-8e2c-4b9c-b74b-474f12957f5a" containerName="revoke-keys" Oct 31 01:29:05.353287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.353269 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.368000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.367967 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:29:05.376766 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.376714 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:29:05.376946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.376817 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:29:05.376946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.376826 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:29:05.379199 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.379171 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn"] Oct 31 01:29:05.473246 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.473211 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vhwnn\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.473438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.473262 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6jlm\" (UniqueName: \"kubernetes.io/projected/e5c489ad-d658-4ff8-90ee-ea39aceede02-kube-api-access-g6jlm\") pod \"maas-api-revoke-sub-0b7e8edb-vhwnn\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.473438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.473280 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vhwnn\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.574855 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.574814 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vhwnn\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.575086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.574885 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6jlm\" (UniqueName: \"kubernetes.io/projected/e5c489ad-d658-4ff8-90ee-ea39aceede02-kube-api-access-g6jlm\") pod \"maas-api-revoke-sub-0b7e8edb-vhwnn\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.575086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.574914 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vhwnn\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.575687 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.575655 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vhwnn\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.577306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.577287 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vhwnn\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.595794 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.595766 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6jlm\" (UniqueName: \"kubernetes.io/projected/e5c489ad-d658-4ff8-90ee-ea39aceede02-kube-api-access-g6jlm\") pod \"maas-api-revoke-sub-0b7e8edb-vhwnn\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.674875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.674778 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:05.973110 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:05.972760 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn"] Oct 31 01:29:05.975071 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:29:05.975038 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c489ad_d658_4ff8_90ee_ea39aceede02.slice/crio-cbf0076fc8b542fe4d893f343e60884240b4a901e6bc93950c1d0508a18a8f08 WatchSource:0}: Error finding container cbf0076fc8b542fe4d893f343e60884240b4a901e6bc93950c1d0508a18a8f08: Status 404 returned error can't find the container with id cbf0076fc8b542fe4d893f343e60884240b4a901e6bc93950c1d0508a18a8f08 Oct 31 01:29:06.472496 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:06.472434 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" event={"ID":"e5c489ad-d658-4ff8-90ee-ea39aceede02","Type":"ContainerStarted","Data":"6d1ee3cd32f106cfe3dffc75c4b7cdd196eebbff5fbef9615e3aa4681bec6639"} Oct 31 01:29:06.472988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:06.472512 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" event={"ID":"e5c489ad-d658-4ff8-90ee-ea39aceede02","Type":"ContainerStarted","Data":"cbf0076fc8b542fe4d893f343e60884240b4a901e6bc93950c1d0508a18a8f08"} Oct 31 01:29:07.479787 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:07.479748 5978 generic.go:356] "Generic (PLEG): container finished" podID="e5c489ad-d658-4ff8-90ee-ea39aceede02" containerID="6d1ee3cd32f106cfe3dffc75c4b7cdd196eebbff5fbef9615e3aa4681bec6639" exitCode=0 Oct 31 01:29:07.480189 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:07.479811 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" event={"ID":"e5c489ad-d658-4ff8-90ee-ea39aceede02","Type":"ContainerDied","Data":"6d1ee3cd32f106cfe3dffc75c4b7cdd196eebbff5fbef9615e3aa4681bec6639"} Oct 31 01:29:08.624997 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.624972 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:08.706050 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.706012 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-openshift-ca\") pod \"e5c489ad-d658-4ff8-90ee-ea39aceede02\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " Oct 31 01:29:08.706239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.706081 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c489ad-d658-4ff8-90ee-ea39aceede02-kube-api-access-g6jlm\" (UniqueName: \"kubernetes.io/projected/e5c489ad-d658-4ff8-90ee-ea39aceede02-kube-api-access-g6jlm\") pod \"e5c489ad-d658-4ff8-90ee-ea39aceede02\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " Oct 31 01:29:08.706239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.706211 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-xks-ca\") pod \"e5c489ad-d658-4ff8-90ee-ea39aceede02\" (UID: \"e5c489ad-d658-4ff8-90ee-ea39aceede02\") " Oct 31 01:29:08.706446 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.706423 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-openshift-ca" pod "e5c489ad-d658-4ff8-90ee-ea39aceede02" (UID: "e5c489ad-d658-4ff8-90ee-ea39aceede02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:08.706628 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.706605 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:29:08.708254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.708229 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-xks-ca" pod "e5c489ad-d658-4ff8-90ee-ea39aceede02" (UID: "e5c489ad-d658-4ff8-90ee-ea39aceede02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:08.708354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.708327 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c489ad-d658-4ff8-90ee-ea39aceede02-kube-api-access-g6jlm" pod "e5c489ad-d658-4ff8-90ee-ea39aceede02" (UID: "e5c489ad-d658-4ff8-90ee-ea39aceede02"). InnerVolumeSpecName "kube-api-access-g6jlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:08.808241 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.808156 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6jlm\" (UniqueName: \"kubernetes.io/projected/e5c489ad-d658-4ff8-90ee-ea39aceede02-kube-api-access-g6jlm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:29:08.808241 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:08.808189 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c489ad-d658-4ff8-90ee-ea39aceede02-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:29:09.492663 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:09.492631 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" Oct 31 01:29:09.492891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:09.492666 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn" event={"ID":"e5c489ad-d658-4ff8-90ee-ea39aceede02","Type":"ContainerDied","Data":"cbf0076fc8b542fe4d893f343e60884240b4a901e6bc93950c1d0508a18a8f08"} Oct 31 01:29:09.492891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:09.492716 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf0076fc8b542fe4d893f343e60884240b4a901e6bc93950c1d0508a18a8f08" Oct 31 01:29:10.028250 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:10.028209 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87"] Oct 31 01:29:10.037664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:10.037632 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28aabcff-gjn87"] Oct 31 01:29:10.424022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:10.423984 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef1d20e-e378-4dd5-9570-6ce1207b3629" path="/var/lib/kubelet/pods/9ef1d20e-e378-4dd5-9570-6ce1207b3629/volumes" Oct 31 01:29:16.120828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.120791 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb"] Oct 31 01:29:16.121620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.121596 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e5c489ad-d658-4ff8-90ee-ea39aceede02" containerName="revoke-keys" Oct 31 01:29:16.121620 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.121622 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c489ad-d658-4ff8-90ee-ea39aceede02" containerName="revoke-keys" Oct 31 01:29:16.121766 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.121705 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c489ad-d658-4ff8-90ee-ea39aceede02" containerName="revoke-keys" Oct 31 01:29:16.124940 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.124918 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.137359 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.137325 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:29:16.146989 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.146952 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:29:16.147205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.147022 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:29:16.147205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.147112 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:29:16.150192 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.150157 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb"] Oct 31 01:29:16.185659 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.185622 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mpgzb\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.185872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.185759 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bzg4\" (UniqueName: \"kubernetes.io/projected/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-kube-api-access-5bzg4\") pod \"maas-api-revoke-sub-3d4156f1-mpgzb\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.185872 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.185844 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mpgzb\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.286841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.286801 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bzg4\" (UniqueName: \"kubernetes.io/projected/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-kube-api-access-5bzg4\") pod \"maas-api-revoke-sub-3d4156f1-mpgzb\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.287059 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.286891 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mpgzb\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.287059 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.286932 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mpgzb\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.287672 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.287644 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mpgzb\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.289403 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.289384 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mpgzb\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.307929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.307896 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bzg4\" (UniqueName: \"kubernetes.io/projected/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-kube-api-access-5bzg4\") pod \"maas-api-revoke-sub-3d4156f1-mpgzb\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.438964 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.438872 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:16.737429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:16.737402 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb"] Oct 31 01:29:16.739927 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:29:16.739896 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76a25a8_34f7_4c05_bdef_48ca6fb7b078.slice/crio-b6ab04ce4c5eba3d41de680b191acf3ec96511a955906ca69d57f57168678893 WatchSource:0}: Error finding container b6ab04ce4c5eba3d41de680b191acf3ec96511a955906ca69d57f57168678893: Status 404 returned error can't find the container with id b6ab04ce4c5eba3d41de680b191acf3ec96511a955906ca69d57f57168678893 Oct 31 01:29:17.542141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:17.542102 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" event={"ID":"b76a25a8-34f7-4c05-bdef-48ca6fb7b078","Type":"ContainerStarted","Data":"456639ca91f7c50c39669dae8ed54065e74b9dad4c1851da966e38ab32f10346"} Oct 31 01:29:17.542638 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:17.542161 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" event={"ID":"b76a25a8-34f7-4c05-bdef-48ca6fb7b078","Type":"ContainerStarted","Data":"b6ab04ce4c5eba3d41de680b191acf3ec96511a955906ca69d57f57168678893"} Oct 31 01:29:17.578792 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:17.578733 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" podStartSLOduration=1.578714045 podStartE2EDuration="1.578714045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:17.577387403 +0000 UTC m=+4175.275501168" watchObservedRunningTime="2026-10-31 01:29:17.578714045 +0000 UTC m=+4175.276827811" Oct 31 01:29:18.549125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:18.549092 5978 generic.go:356] "Generic (PLEG): container finished" podID="b76a25a8-34f7-4c05-bdef-48ca6fb7b078" containerID="456639ca91f7c50c39669dae8ed54065e74b9dad4c1851da966e38ab32f10346" exitCode=0 Oct 31 01:29:18.549631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:18.549186 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" event={"ID":"b76a25a8-34f7-4c05-bdef-48ca6fb7b078","Type":"ContainerDied","Data":"456639ca91f7c50c39669dae8ed54065e74b9dad4c1851da966e38ab32f10346"} Oct 31 01:29:19.697782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.697755 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:19.826801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.826766 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-xks-ca\") pod \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " Oct 31 01:29:19.827003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.826820 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-kube-api-access-5bzg4\" (UniqueName: \"kubernetes.io/projected/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-kube-api-access-5bzg4\") pod \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " Oct 31 01:29:19.827003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.826953 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-openshift-ca\") pod \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\" (UID: \"b76a25a8-34f7-4c05-bdef-48ca6fb7b078\") " Oct 31 01:29:19.827298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.827272 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-openshift-ca" pod "b76a25a8-34f7-4c05-bdef-48ca6fb7b078" (UID: "b76a25a8-34f7-4c05-bdef-48ca6fb7b078"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:19.828961 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.828936 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-xks-ca" pod "b76a25a8-34f7-4c05-bdef-48ca6fb7b078" (UID: "b76a25a8-34f7-4c05-bdef-48ca6fb7b078"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:19.829091 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.829072 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-kube-api-access-5bzg4" pod "b76a25a8-34f7-4c05-bdef-48ca6fb7b078" (UID: "b76a25a8-34f7-4c05-bdef-48ca6fb7b078"). InnerVolumeSpecName "kube-api-access-5bzg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:19.928229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.928191 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:29:19.928229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.928225 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:29:19.928229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:19.928235 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bzg4\" (UniqueName: \"kubernetes.io/projected/b76a25a8-34f7-4c05-bdef-48ca6fb7b078-kube-api-access-5bzg4\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:29:20.561490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:20.561432 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" event={"ID":"b76a25a8-34f7-4c05-bdef-48ca6fb7b078","Type":"ContainerDied","Data":"b6ab04ce4c5eba3d41de680b191acf3ec96511a955906ca69d57f57168678893"} Oct 31 01:29:20.561674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:20.561512 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6ab04ce4c5eba3d41de680b191acf3ec96511a955906ca69d57f57168678893" Oct 31 01:29:20.561674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:20.561448 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb" Oct 31 01:29:30.601992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:30.601606 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6bxb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:29:30.601992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:30.601690 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" podUID="694e2b84-5e34-4d1e-bf99-16b261a95272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:29:30.612651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:30.612606 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ldf84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:29:30.612956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:30.612923 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" podUID="95964f90-7a0d-4fe8-97aa-6f10cf619a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:29:33.303851 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:29:33.303806 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:29:37.537908 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.537871 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb"] Oct 31 01:29:37.547021 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.546989 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mpgzb"] Oct 31 01:29:37.586502 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.586470 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc"] Oct 31 01:29:37.587265 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.587239 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b76a25a8-34f7-4c05-bdef-48ca6fb7b078" containerName="revoke-keys" Oct 31 01:29:37.587265 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.587264 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76a25a8-34f7-4c05-bdef-48ca6fb7b078" containerName="revoke-keys" Oct 31 01:29:37.587421 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.587373 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76a25a8-34f7-4c05-bdef-48ca6fb7b078" containerName="revoke-keys" Oct 31 01:29:37.590734 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.590715 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.603179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.603125 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:29:37.603362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.603125 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:29:37.603362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.603242 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:29:37.603362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.603284 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:29:37.615322 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.615289 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc"] Oct 31 01:29:37.720979 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.720935 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswkc\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.721198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.721103 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswkc\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.721198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.721156 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hql8g\" (UniqueName: \"kubernetes.io/projected/b4d36c59-48e4-4047-9ccb-798b086bf50c-kube-api-access-hql8g\") pod \"maas-api-revoke-sub-3d4156f1-vswkc\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.822057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.822016 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswkc\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.822272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.822090 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hql8g\" (UniqueName: \"kubernetes.io/projected/b4d36c59-48e4-4047-9ccb-798b086bf50c-kube-api-access-hql8g\") pod \"maas-api-revoke-sub-3d4156f1-vswkc\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.822272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.822137 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswkc\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.822849 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.822817 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswkc\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.824353 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.824327 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswkc\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.843363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.843334 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hql8g\" (UniqueName: \"kubernetes.io/projected/b4d36c59-48e4-4047-9ccb-798b086bf50c-kube-api-access-hql8g\") pod \"maas-api-revoke-sub-3d4156f1-vswkc\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:37.907252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:37.907212 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:38.205229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:38.205203 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc"] Oct 31 01:29:38.207839 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:29:38.207811 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d36c59_48e4_4047_9ccb_798b086bf50c.slice/crio-4f6f0de0b5cd22f5c135076c85cf003a6e951218ca4ba93d2bab628fdc578bd3 WatchSource:0}: Error finding container 4f6f0de0b5cd22f5c135076c85cf003a6e951218ca4ba93d2bab628fdc578bd3: Status 404 returned error can't find the container with id 4f6f0de0b5cd22f5c135076c85cf003a6e951218ca4ba93d2bab628fdc578bd3 Oct 31 01:29:38.425488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:38.425438 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76a25a8-34f7-4c05-bdef-48ca6fb7b078" path="/var/lib/kubelet/pods/b76a25a8-34f7-4c05-bdef-48ca6fb7b078/volumes" Oct 31 01:29:38.702175 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:38.702143 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" event={"ID":"b4d36c59-48e4-4047-9ccb-798b086bf50c","Type":"ContainerStarted","Data":"5291026b01c8c21bc6c65025079a3988ac597fc41b075adb223b558ebe42acf6"} Oct 31 01:29:38.702740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:38.702206 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" event={"ID":"b4d36c59-48e4-4047-9ccb-798b086bf50c","Type":"ContainerStarted","Data":"4f6f0de0b5cd22f5c135076c85cf003a6e951218ca4ba93d2bab628fdc578bd3"} Oct 31 01:29:38.738815 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:38.738749 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" podStartSLOduration=1.7387292570000001 podStartE2EDuration="1.738729257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:38.738319942 +0000 UTC m=+4196.436433706" watchObservedRunningTime="2026-10-31 01:29:38.738729257 +0000 UTC m=+4196.436843026" Oct 31 01:29:39.710494 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:39.710435 5978 generic.go:356] "Generic (PLEG): container finished" podID="b4d36c59-48e4-4047-9ccb-798b086bf50c" containerID="5291026b01c8c21bc6c65025079a3988ac597fc41b075adb223b558ebe42acf6" exitCode=0 Oct 31 01:29:39.710935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:39.710518 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" event={"ID":"b4d36c59-48e4-4047-9ccb-798b086bf50c","Type":"ContainerDied","Data":"5291026b01c8c21bc6c65025079a3988ac597fc41b075adb223b558ebe42acf6"} Oct 31 01:29:40.866554 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:40.866522 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:40.954826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:40.954791 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-openshift-ca\") pod \"b4d36c59-48e4-4047-9ccb-798b086bf50c\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " Oct 31 01:29:40.955020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:40.954846 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-xks-ca\") pod \"b4d36c59-48e4-4047-9ccb-798b086bf50c\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " Oct 31 01:29:40.955020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:40.954899 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d36c59-48e4-4047-9ccb-798b086bf50c-kube-api-access-hql8g\" (UniqueName: \"kubernetes.io/projected/b4d36c59-48e4-4047-9ccb-798b086bf50c-kube-api-access-hql8g\") pod \"b4d36c59-48e4-4047-9ccb-798b086bf50c\" (UID: \"b4d36c59-48e4-4047-9ccb-798b086bf50c\") " Oct 31 01:29:40.955184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:40.955151 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-openshift-ca" pod "b4d36c59-48e4-4047-9ccb-798b086bf50c" (UID: "b4d36c59-48e4-4047-9ccb-798b086bf50c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:40.955297 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:40.955284 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:29:40.957083 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:40.957056 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d36c59-48e4-4047-9ccb-798b086bf50c-kube-api-access-hql8g" pod "b4d36c59-48e4-4047-9ccb-798b086bf50c" (UID: "b4d36c59-48e4-4047-9ccb-798b086bf50c"). InnerVolumeSpecName "kube-api-access-hql8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:40.957169 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:40.957103 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-xks-ca" pod "b4d36c59-48e4-4047-9ccb-798b086bf50c" (UID: "b4d36c59-48e4-4047-9ccb-798b086bf50c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:41.027091 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:41.026971 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn"] Oct 31 01:29:41.030485 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:41.030429 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d4260a-g9xsn"] Oct 31 01:29:41.055985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:41.055951 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hql8g\" (UniqueName: \"kubernetes.io/projected/b4d36c59-48e4-4047-9ccb-798b086bf50c-kube-api-access-hql8g\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:29:41.055985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:41.055984 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d36c59-48e4-4047-9ccb-798b086bf50c-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:29:41.723928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:41.723890 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" event={"ID":"b4d36c59-48e4-4047-9ccb-798b086bf50c","Type":"ContainerDied","Data":"4f6f0de0b5cd22f5c135076c85cf003a6e951218ca4ba93d2bab628fdc578bd3"} Oct 31 01:29:41.724128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:41.723951 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f6f0de0b5cd22f5c135076c85cf003a6e951218ca4ba93d2bab628fdc578bd3" Oct 31 01:29:41.724128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:41.723902 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc" Oct 31 01:29:42.424055 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:42.424022 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d8d172-d451-41de-bf5d-fd5af6482b03" path="/var/lib/kubelet/pods/46d8d172-d451-41de-bf5d-fd5af6482b03/volumes" Oct 31 01:29:49.194722 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:49.194692 5978 scope.go:122] "RemoveContainer" containerID="363c69ea0f656b9eda5f0d21ae579d407f992addb189f40d51e59b23432d898e" Oct 31 01:29:49.204180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:49.204156 5978 scope.go:122] "RemoveContainer" containerID="6e0c8c0c06c4302bb47852918c2d017c571799c19dedec49b5a1421fc46b0e13" Oct 31 01:29:50.731436 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:50.731404 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:29:50.741974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:50.741948 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:29:51.693464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:29:51.693420 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:30:00.139128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.139084 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8"] Oct 31 01:30:00.139788 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.139766 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b4d36c59-48e4-4047-9ccb-798b086bf50c" containerName="revoke-keys" Oct 31 01:30:00.139883 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.139790 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d36c59-48e4-4047-9ccb-798b086bf50c" containerName="revoke-keys" Oct 31 01:30:00.139954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.139940 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d36c59-48e4-4047-9ccb-798b086bf50c" containerName="revoke-keys" Oct 31 01:30:00.143229 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.143205 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:00.156373 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.156350 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:30:00.165766 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.165728 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:30:00.165948 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.165806 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:30:00.169165 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.169136 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8"] Oct 31 01:30:00.171483 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.171437 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ea11c-7e3f-43ed-bfee-b937049d6f23-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-27pc8\" (UID: \"073ea11c-7e3f-43ed-bfee-b937049d6f23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:00.171647 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.171620 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncs7\" (UniqueName: \"kubernetes.io/projected/073ea11c-7e3f-43ed-bfee-b937049d6f23-kube-api-access-gncs7\") pod \"maas-api-key-cleanup-29890170-27pc8\" (UID: \"073ea11c-7e3f-43ed-bfee-b937049d6f23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:00.272560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.272512 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ea11c-7e3f-43ed-bfee-b937049d6f23-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-27pc8\" (UID: \"073ea11c-7e3f-43ed-bfee-b937049d6f23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:00.272795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.272679 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncs7\" (UniqueName: \"kubernetes.io/projected/073ea11c-7e3f-43ed-bfee-b937049d6f23-kube-api-access-gncs7\") pod \"maas-api-key-cleanup-29890170-27pc8\" (UID: \"073ea11c-7e3f-43ed-bfee-b937049d6f23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:00.273243 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.273215 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ea11c-7e3f-43ed-bfee-b937049d6f23-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-27pc8\" (UID: \"073ea11c-7e3f-43ed-bfee-b937049d6f23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:00.398129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.398036 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncs7\" (UniqueName: \"kubernetes.io/projected/073ea11c-7e3f-43ed-bfee-b937049d6f23-kube-api-access-gncs7\") pod \"maas-api-key-cleanup-29890170-27pc8\" (UID: \"073ea11c-7e3f-43ed-bfee-b937049d6f23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:00.457511 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.457445 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:00.757171 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.757089 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8"] Oct 31 01:30:00.760674 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:30:00.760638 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073ea11c_7e3f_43ed_bfee_b937049d6f23.slice/crio-1daf2c4442fc7dbc545649a5501845109c5891c9a0f637b842435f77095342ca WatchSource:0}: Error finding container 1daf2c4442fc7dbc545649a5501845109c5891c9a0f637b842435f77095342ca: Status 404 returned error can't find the container with id 1daf2c4442fc7dbc545649a5501845109c5891c9a0f637b842435f77095342ca Oct 31 01:30:00.834604 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:00.834569 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" event={"ID":"073ea11c-7e3f-43ed-bfee-b937049d6f23","Type":"ContainerStarted","Data":"1daf2c4442fc7dbc545649a5501845109c5891c9a0f637b842435f77095342ca"} Oct 31 01:30:01.843006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:01.842968 5978 generic.go:356] "Generic (PLEG): container finished" podID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerID="df14cce188f0e3211f0ea3f84c20546cf093d38da1e04145ba4c3acdc6e4b065" exitCode=60 Oct 31 01:30:01.843558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:01.843052 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" event={"ID":"073ea11c-7e3f-43ed-bfee-b937049d6f23","Type":"ContainerDied","Data":"df14cce188f0e3211f0ea3f84c20546cf093d38da1e04145ba4c3acdc6e4b065"} Oct 31 01:30:01.843558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:01.843306 5978 scope.go:122] "RemoveContainer" containerID="df14cce188f0e3211f0ea3f84c20546cf093d38da1e04145ba4c3acdc6e4b065" Oct 31 01:30:02.854658 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:02.854624 5978 generic.go:356] "Generic (PLEG): container finished" podID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerID="4083efe5ff8e8ed5b19b0bcf6154cf919f42e3c80ef8411fd784c402b2789d50" exitCode=60 Oct 31 01:30:02.855109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:02.854711 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" event={"ID":"073ea11c-7e3f-43ed-bfee-b937049d6f23","Type":"ContainerDied","Data":"4083efe5ff8e8ed5b19b0bcf6154cf919f42e3c80ef8411fd784c402b2789d50"} Oct 31 01:30:02.855109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:02.854773 5978 scope.go:122] "RemoveContainer" containerID="df14cce188f0e3211f0ea3f84c20546cf093d38da1e04145ba4c3acdc6e4b065" Oct 31 01:30:02.855109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:02.854957 5978 scope.go:122] "RemoveContainer" containerID="4083efe5ff8e8ed5b19b0bcf6154cf919f42e3c80ef8411fd784c402b2789d50" Oct 31 01:30:02.855302 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:30:02.855230 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-27pc8_odh-ai-gateway-infra(073ea11c-7e3f-43ed-bfee-b937049d6f23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" Oct 31 01:30:03.182317 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:30:03.182272 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:30:03.862167 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:03.862143 5978 scope.go:122] "RemoveContainer" containerID="4083efe5ff8e8ed5b19b0bcf6154cf919f42e3c80ef8411fd784c402b2789d50" Oct 31 01:30:03.862653 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:30:03.862405 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-27pc8_odh-ai-gateway-infra(073ea11c-7e3f-43ed-bfee-b937049d6f23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" Oct 31 01:30:15.418153 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:15.418114 5978 scope.go:122] "RemoveContainer" containerID="4083efe5ff8e8ed5b19b0bcf6154cf919f42e3c80ef8411fd784c402b2789d50" Oct 31 01:30:15.960179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:15.960144 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" event={"ID":"073ea11c-7e3f-43ed-bfee-b937049d6f23","Type":"ContainerStarted","Data":"044259f6dbbc2b6eb0873d7c7e6e56c8c7216a496359baf9886a4918e516d1e4"} Oct 31 01:30:15.997372 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:15.997257 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" podStartSLOduration=15.997239477 podStartE2EDuration="15.997239477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:15.995007644 +0000 UTC m=+4233.693121413" watchObservedRunningTime="2026-10-31 01:30:15.997239477 +0000 UTC m=+4233.695353241" Oct 31 01:30:16.967480 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:16.967414 5978 generic.go:356] "Generic (PLEG): container finished" podID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerID="044259f6dbbc2b6eb0873d7c7e6e56c8c7216a496359baf9886a4918e516d1e4" exitCode=60 Oct 31 01:30:16.967974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:16.967503 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" event={"ID":"073ea11c-7e3f-43ed-bfee-b937049d6f23","Type":"ContainerDied","Data":"044259f6dbbc2b6eb0873d7c7e6e56c8c7216a496359baf9886a4918e516d1e4"} Oct 31 01:30:16.967974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:16.967571 5978 scope.go:122] "RemoveContainer" containerID="4083efe5ff8e8ed5b19b0bcf6154cf919f42e3c80ef8411fd784c402b2789d50" Oct 31 01:30:16.967974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:16.967929 5978 scope.go:122] "RemoveContainer" containerID="044259f6dbbc2b6eb0873d7c7e6e56c8c7216a496359baf9886a4918e516d1e4" Oct 31 01:30:16.968253 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:30:16.968231 5978 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-27pc8_odh-ai-gateway-infra(073ea11c-7e3f-43ed-bfee-b937049d6f23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" Oct 31 01:30:17.005362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:17.005326 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8"] Oct 31 01:30:18.114704 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.114673 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:18.196575 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.196541 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073ea11c-7e3f-43ed-bfee-b937049d6f23-kube-api-access-gncs7\" (UniqueName: \"kubernetes.io/projected/073ea11c-7e3f-43ed-bfee-b937049d6f23-kube-api-access-gncs7\") pod \"073ea11c-7e3f-43ed-bfee-b937049d6f23\" (UID: \"073ea11c-7e3f-43ed-bfee-b937049d6f23\") " Oct 31 01:30:18.196784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.196598 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073ea11c-7e3f-43ed-bfee-b937049d6f23-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ea11c-7e3f-43ed-bfee-b937049d6f23-maas-api-service-ca\") pod \"073ea11c-7e3f-43ed-bfee-b937049d6f23\" (UID: \"073ea11c-7e3f-43ed-bfee-b937049d6f23\") " Oct 31 01:30:18.197011 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.196989 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073ea11c-7e3f-43ed-bfee-b937049d6f23-maas-api-service-ca" pod "073ea11c-7e3f-43ed-bfee-b937049d6f23" (UID: "073ea11c-7e3f-43ed-bfee-b937049d6f23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:18.198868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.198841 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073ea11c-7e3f-43ed-bfee-b937049d6f23-kube-api-access-gncs7" pod "073ea11c-7e3f-43ed-bfee-b937049d6f23" (UID: "073ea11c-7e3f-43ed-bfee-b937049d6f23"). InnerVolumeSpecName "kube-api-access-gncs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:18.297743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.297646 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073ea11c-7e3f-43ed-bfee-b937049d6f23-maas-api-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:30:18.297743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.297681 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gncs7\" (UniqueName: \"kubernetes.io/projected/073ea11c-7e3f-43ed-bfee-b937049d6f23-kube-api-access-gncs7\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:30:18.982759 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.982724 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" Oct 31 01:30:18.982971 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.982761 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8" event={"ID":"073ea11c-7e3f-43ed-bfee-b937049d6f23","Type":"ContainerDied","Data":"1daf2c4442fc7dbc545649a5501845109c5891c9a0f637b842435f77095342ca"} Oct 31 01:30:18.982971 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:18.982825 5978 scope.go:122] "RemoveContainer" containerID="044259f6dbbc2b6eb0873d7c7e6e56c8c7216a496359baf9886a4918e516d1e4" Oct 31 01:30:19.026936 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:19.026887 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8"] Oct 31 01:30:19.036609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:19.036572 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-27pc8"] Oct 31 01:30:20.424793 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:20.424748 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" path="/var/lib/kubelet/pods/073ea11c-7e3f-43ed-bfee-b937049d6f23/volumes" Oct 31 01:30:21.968716 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.968675 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr"] Oct 31 01:30:21.969241 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.969221 5978 cpu_manager.go:414] "containerMap: removing container" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerName="cleanup" Oct 31 01:30:21.969298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.969243 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerName="cleanup" Oct 31 01:30:21.969298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.969252 5978 cpu_manager.go:414] "containerMap: removing container" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerName="cleanup" Oct 31 01:30:21.969298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.969258 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerName="cleanup" Oct 31 01:30:21.969414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.969360 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerName="cleanup" Oct 31 01:30:21.969414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.969368 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerName="cleanup" Oct 31 01:30:21.972610 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.972594 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:21.984970 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.984946 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:30:21.994651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.994621 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:30:21.994651 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.994627 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:30:21.994812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:21.994653 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:30:22.003642 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.003615 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr"] Oct 31 01:30:22.036009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.035967 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svprw\" (UniqueName: \"kubernetes.io/projected/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-kube-api-access-svprw\") pod \"maas-api-revoke-sub-5a29efb3-nchcr\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.036211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.036023 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-nchcr\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.036211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.036107 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-nchcr\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.136858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.136818 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svprw\" (UniqueName: \"kubernetes.io/projected/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-kube-api-access-svprw\") pod \"maas-api-revoke-sub-5a29efb3-nchcr\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.136858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.136857 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-nchcr\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.137083 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.136882 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-nchcr\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.137605 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.137576 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-nchcr\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.139271 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.139250 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-nchcr\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.157869 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.157841 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svprw\" (UniqueName: \"kubernetes.io/projected/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-kube-api-access-svprw\") pod \"maas-api-revoke-sub-5a29efb3-nchcr\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.285671 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.285594 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:22.589036 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:22.589010 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr"] Oct 31 01:30:22.591212 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:30:22.591182 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaba58e4_a2fd_4fe1_94d2_666fa52ae283.slice/crio-44126afe5a4034ebcb4fea0eb01f050fb8ddf12be688ec3f9e26bc506ee4606a WatchSource:0}: Error finding container 44126afe5a4034ebcb4fea0eb01f050fb8ddf12be688ec3f9e26bc506ee4606a: Status 404 returned error can't find the container with id 44126afe5a4034ebcb4fea0eb01f050fb8ddf12be688ec3f9e26bc506ee4606a Oct 31 01:30:23.010667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:23.010622 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" event={"ID":"eaba58e4-a2fd-4fe1-94d2-666fa52ae283","Type":"ContainerStarted","Data":"f070d00bedf9428f4e95b2986e2ecb5f71822f898f5149c6e46155da09ce65aa"} Oct 31 01:30:23.011159 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:23.010740 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" event={"ID":"eaba58e4-a2fd-4fe1-94d2-666fa52ae283","Type":"ContainerStarted","Data":"44126afe5a4034ebcb4fea0eb01f050fb8ddf12be688ec3f9e26bc506ee4606a"} Oct 31 01:30:23.047014 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:23.046956 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" podStartSLOduration=2.046943072 podStartE2EDuration="2.046943072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:23.046186919 +0000 UTC m=+4240.744300685" watchObservedRunningTime="2026-10-31 01:30:23.046943072 +0000 UTC m=+4240.745056837" Oct 31 01:30:24.023020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:24.022875 5978 generic.go:356] "Generic (PLEG): container finished" podID="eaba58e4-a2fd-4fe1-94d2-666fa52ae283" containerID="f070d00bedf9428f4e95b2986e2ecb5f71822f898f5149c6e46155da09ce65aa" exitCode=0 Oct 31 01:30:24.023020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:24.022941 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" event={"ID":"eaba58e4-a2fd-4fe1-94d2-666fa52ae283","Type":"ContainerDied","Data":"f070d00bedf9428f4e95b2986e2ecb5f71822f898f5149c6e46155da09ce65aa"} Oct 31 01:30:25.945065 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:25.944665 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:26.068542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.067371 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-openshift-ca\") pod \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " Oct 31 01:30:26.068542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.067487 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-xks-ca\") pod \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " Oct 31 01:30:26.068542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.067571 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-kube-api-access-svprw\" (UniqueName: \"kubernetes.io/projected/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-kube-api-access-svprw\") pod \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\" (UID: \"eaba58e4-a2fd-4fe1-94d2-666fa52ae283\") " Oct 31 01:30:26.112633 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.112510 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-kube-api-access-svprw" pod "eaba58e4-a2fd-4fe1-94d2-666fa52ae283" (UID: "eaba58e4-a2fd-4fe1-94d2-666fa52ae283"). InnerVolumeSpecName "kube-api-access-svprw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:26.121778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.121702 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-xks-ca" pod "eaba58e4-a2fd-4fe1-94d2-666fa52ae283" (UID: "eaba58e4-a2fd-4fe1-94d2-666fa52ae283"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:26.137932 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.137620 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" event={"ID":"eaba58e4-a2fd-4fe1-94d2-666fa52ae283","Type":"ContainerDied","Data":"44126afe5a4034ebcb4fea0eb01f050fb8ddf12be688ec3f9e26bc506ee4606a"} Oct 31 01:30:26.138185 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.138165 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44126afe5a4034ebcb4fea0eb01f050fb8ddf12be688ec3f9e26bc506ee4606a" Oct 31 01:30:26.138472 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.138407 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr" Oct 31 01:30:26.175477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.165671 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-openshift-ca" pod "eaba58e4-a2fd-4fe1-94d2-666fa52ae283" (UID: "eaba58e4-a2fd-4fe1-94d2-666fa52ae283"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:26.185899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.185794 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:30:26.185899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.185837 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:30:26.185899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:26.185854 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svprw\" (UniqueName: \"kubernetes.io/projected/eaba58e4-a2fd-4fe1-94d2-666fa52ae283-kube-api-access-svprw\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:30:29.028067 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:29.028025 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn"] Oct 31 01:30:29.037096 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:29.037062 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-k9cxn"] Oct 31 01:30:30.424066 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:30.424022 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1623e9a-d6d5-48fc-a93a-678d36aad715" path="/var/lib/kubelet/pods/d1623e9a-d6d5-48fc-a93a-678d36aad715/volumes" Oct 31 01:30:33.182288 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:30:33.182250 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:30:41.710318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.710276 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc"] Oct 31 01:30:41.719484 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.719436 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswkc"] Oct 31 01:30:41.748218 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.748180 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf"] Oct 31 01:30:41.748906 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.748882 5978 cpu_manager.go:414] "containerMap: removing container" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerName="cleanup" Oct 31 01:30:41.748906 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.748906 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerName="cleanup" Oct 31 01:30:41.749070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.748925 5978 cpu_manager.go:414] "containerMap: removing container" podUID="eaba58e4-a2fd-4fe1-94d2-666fa52ae283" containerName="revoke-keys" Oct 31 01:30:41.749070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.748931 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaba58e4-a2fd-4fe1-94d2-666fa52ae283" containerName="revoke-keys" Oct 31 01:30:41.749070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.749004 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="073ea11c-7e3f-43ed-bfee-b937049d6f23" containerName="cleanup" Oct 31 01:30:41.749070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.749015 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaba58e4-a2fd-4fe1-94d2-666fa52ae283" containerName="revoke-keys" Oct 31 01:30:41.753421 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.753398 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:41.753853 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.753828 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-kube-api-access-fcdzt\") pod \"maas-api-revoke-sub-3d4156f1-9fgmf\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:41.754033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.754013 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fgmf\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:41.754160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.754140 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fgmf\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:41.766000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.765969 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:30:41.766000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.765983 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:30:41.766216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.766005 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:30:41.766216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.766044 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:30:41.773039 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.772999 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf"] Oct 31 01:30:41.854926 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.854889 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fgmf\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:41.855115 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.854940 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-kube-api-access-fcdzt\") pod \"maas-api-revoke-sub-3d4156f1-9fgmf\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:41.855115 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.855046 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fgmf\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:41.855937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.855904 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fgmf\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:41.857416 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.857395 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fgmf\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:41.876245 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:41.876202 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-kube-api-access-fcdzt\") pod \"maas-api-revoke-sub-3d4156f1-9fgmf\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:42.067253 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:42.067221 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:42.364216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:42.363872 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf"] Oct 31 01:30:42.366441 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:30:42.366407 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981d5ad5_2fc1_4a43_bd89_20f1711d5e6f.slice/crio-886d388c3fff296b60ca7acaa864622612f149c19ca6cbf393510cbecf622c6f WatchSource:0}: Error finding container 886d388c3fff296b60ca7acaa864622612f149c19ca6cbf393510cbecf622c6f: Status 404 returned error can't find the container with id 886d388c3fff296b60ca7acaa864622612f149c19ca6cbf393510cbecf622c6f Oct 31 01:30:42.424581 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:42.424547 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d36c59-48e4-4047-9ccb-798b086bf50c" path="/var/lib/kubelet/pods/b4d36c59-48e4-4047-9ccb-798b086bf50c/volumes" Oct 31 01:30:43.243118 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:43.243085 5978 generic.go:356] "Generic (PLEG): container finished" podID="981d5ad5-2fc1-4a43-bd89-20f1711d5e6f" containerID="0604cae334d75c796e32dbe1e5aa7c5ea77ad14b4f304374e8cd0b683189da58" exitCode=0 Oct 31 01:30:43.243548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:43.243153 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" event={"ID":"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f","Type":"ContainerDied","Data":"0604cae334d75c796e32dbe1e5aa7c5ea77ad14b4f304374e8cd0b683189da58"} Oct 31 01:30:43.243548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:43.243190 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" event={"ID":"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f","Type":"ContainerStarted","Data":"886d388c3fff296b60ca7acaa864622612f149c19ca6cbf393510cbecf622c6f"} Oct 31 01:30:44.391920 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.391890 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:44.492139 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.492099 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-openshift-ca\") pod \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " Oct 31 01:30:44.492336 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.492211 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-xks-ca\") pod \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " Oct 31 01:30:44.492336 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.492239 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-kube-api-access-fcdzt\") pod \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\" (UID: \"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f\") " Oct 31 01:30:44.492575 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.492546 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-openshift-ca" pod "981d5ad5-2fc1-4a43-bd89-20f1711d5e6f" (UID: "981d5ad5-2fc1-4a43-bd89-20f1711d5e6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:44.492864 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.492837 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:30:44.494515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.494485 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-kube-api-access-fcdzt" pod "981d5ad5-2fc1-4a43-bd89-20f1711d5e6f" (UID: "981d5ad5-2fc1-4a43-bd89-20f1711d5e6f"). InnerVolumeSpecName "kube-api-access-fcdzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:44.495078 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.494542 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-xks-ca" pod "981d5ad5-2fc1-4a43-bd89-20f1711d5e6f" (UID: "981d5ad5-2fc1-4a43-bd89-20f1711d5e6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:44.594287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.594245 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:30:44.594287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:44.594280 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f-kube-api-access-fcdzt\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:30:45.257501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:45.257440 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" Oct 31 01:30:45.257703 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:45.257498 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf" event={"ID":"981d5ad5-2fc1-4a43-bd89-20f1711d5e6f","Type":"ContainerDied","Data":"886d388c3fff296b60ca7acaa864622612f149c19ca6cbf393510cbecf622c6f"} Oct 31 01:30:45.257703 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:45.257547 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886d388c3fff296b60ca7acaa864622612f149c19ca6cbf393510cbecf622c6f" Oct 31 01:30:49.234856 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:49.234815 5978 scope.go:122] "RemoveContainer" containerID="687d720c978f7a700bbec562e9b67e0d5e7da962e8df5602b9631dbabe846c9b" Oct 31 01:30:49.244615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:49.244590 5978 scope.go:122] "RemoveContainer" containerID="a60cec6cffea7d0bf52fa3fc43ea4bb772a0bbbbdc8e0ff5bc8c5fbec6126134" Oct 31 01:30:49.256437 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:49.256402 5978 scope.go:122] "RemoveContainer" containerID="81ba9e730b576c487be4b0c49377700c63b5363693f8ff3418238ac9fb5f34a9" Oct 31 01:30:51.741481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:51.741415 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:30:53.264538 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:53.264502 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:30:53.984754 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:53.984689 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:30:54.129927 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:30:54.129892 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:03.182109 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:31:03.182068 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:31:03.695846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:03.695810 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:04.232565 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:04.232525 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:04.378342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:04.378294 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:07.931726 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:07.931677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:08.115092 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:08.115055 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:09.240446 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.240404 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz"] Oct 31 01:31:09.240954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.240937 5978 cpu_manager.go:414] "containerMap: removing container" podUID="981d5ad5-2fc1-4a43-bd89-20f1711d5e6f" containerName="revoke-keys" Oct 31 01:31:09.241004 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.240956 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981d5ad5-2fc1-4a43-bd89-20f1711d5e6f" containerName="revoke-keys" Oct 31 01:31:09.241046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.241020 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="981d5ad5-2fc1-4a43-bd89-20f1711d5e6f" containerName="revoke-keys" Oct 31 01:31:09.244392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.244376 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.269060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.269023 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-e8e612-serving-cert\"" Oct 31 01:31:09.269060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.269022 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:31:09.269060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.269054 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:31:09.269389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.269235 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:31:09.269389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.269325 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cckh7\"" Oct 31 01:31:09.273465 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.273420 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz"] Oct 31 01:31:09.334860 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.334825 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.335042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.334880 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28d1687-0830-480f-8189-277879542db1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.335042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.335007 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-979lw\" (UniqueName: \"kubernetes.io/projected/b28d1687-0830-480f-8189-277879542db1-kube-api-access-979lw\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.335042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.335035 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.388626 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.388589 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:09.435913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.435883 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.436109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.435933 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28d1687-0830-480f-8189-277879542db1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.436109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.435990 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-979lw\" (UniqueName: \"kubernetes.io/projected/b28d1687-0830-480f-8189-277879542db1-kube-api-access-979lw\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.436109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.436013 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.436755 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.436731 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28d1687-0830-480f-8189-277879542db1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.438490 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.438469 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.438586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.438529 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.456694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.456664 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-979lw\" (UniqueName: \"kubernetes.io/projected/b28d1687-0830-480f-8189-277879542db1-kube-api-access-979lw\") pod \"maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.553123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.553022 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:09.559923 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.559887 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:09.709515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.709483 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:09.868995 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.868270 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz"] Oct 31 01:31:09.870517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.870244 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:09.871584 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:31:09.871555 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28d1687_0830_480f_8189_277879542db1.slice/crio-94be194da30eddfbd64691ece0278a8142157f718530f460313df3b973a64432 WatchSource:0}: Error finding container 94be194da30eddfbd64691ece0278a8142157f718530f460313df3b973a64432: Status 404 returned error can't find the container with id 94be194da30eddfbd64691ece0278a8142157f718530f460313df3b973a64432 Oct 31 01:31:09.873031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:09.873012 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:31:10.434667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:10.434632 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:10.435161 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:10.434878 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" event={"ID":"b28d1687-0830-480f-8189-277879542db1","Type":"ContainerStarted","Data":"a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0"} Oct 31 01:31:10.435161 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:10.434917 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" event={"ID":"b28d1687-0830-480f-8189-277879542db1","Type":"ContainerStarted","Data":"94be194da30eddfbd64691ece0278a8142157f718530f460313df3b973a64432"} Oct 31 01:31:10.468807 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:10.468750 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" podStartSLOduration=1.033803812 podStartE2EDuration="1.468735722s" totalImagesPullingTime="434.93191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:10.467600329 +0000 UTC m=+4288.165714095" watchObservedRunningTime="2026-10-31 01:31:10.468735722 +0000 UTC m=+4288.166849484" Oct 31 01:31:11.201946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:11.201905 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:12.850737 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:12.850700 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:14.621478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:14.621424 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:16.432429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:16.432402 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:31:16.444996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:16.444962 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:18.155738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:18.155701 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:19.898016 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:19.897986 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:21.501382 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:21.501347 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:22.903996 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:22.903946 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:23.125564 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:23.125522 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:23.628389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:23.628353 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:23.951122 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:23.951028 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:24.802159 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.802114 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:24.915128 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.915089 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p"] Oct 31 01:31:24.919042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.919019 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:24.933525 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.933493 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e8e612\"/\"default-dockercfg-mckss\"" Oct 31 01:31:24.935981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.935957 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e8e612\"/\"openshift-service-ca.crt\"" Oct 31 01:31:24.936125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.935997 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e8e612\"/\"kube-root-ca.crt\"" Oct 31 01:31:24.945829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.945802 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p"] Oct 31 01:31:24.999383 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.999349 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-dshm\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:24.999867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.999397 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tls-certs\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:24.999867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.999417 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-model-cache\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:24.999867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.999517 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcx6v\" (UniqueName: \"kubernetes.io/projected/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-kube-api-access-zcx6v\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:24.999867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.999613 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tmp-dir\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:24.999867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:24.999665 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-home\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.036102 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.036073 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e8e612\"/\"simulator-single-main-e8e612-kserve-self-signed-certs\"" Oct 31 01:31:25.040738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.040702 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:25.100858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.100823 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-dshm\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.101074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.100880 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tls-certs\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.101074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.100899 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-model-cache\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.101074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.100919 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcx6v\" (UniqueName: \"kubernetes.io/projected/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-kube-api-access-zcx6v\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.101074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.100947 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tmp-dir\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.101074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.100978 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-home\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.101928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.101381 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-model-cache\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.101928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.101404 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tmp-dir\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.101928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.101443 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-home\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.103837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.103807 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-dshm\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.104063 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.104048 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tls-certs\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.129643 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.129606 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcx6v\" (UniqueName: \"kubernetes.io/projected/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-kube-api-access-zcx6v\") pod \"simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.235546 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.235509 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:25.302077 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.299975 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:25.534954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:25.534927 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p"] Oct 31 01:31:25.536417 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:31:25.536388 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31f02d62_d6ed_40bb_9e8b_d172d5a7d522.slice/crio-682bf84481e5a8e50b5a937ae9ebec44faf4eaddb37f6d5da0d92660edf465f4 WatchSource:0}: Error finding container 682bf84481e5a8e50b5a937ae9ebec44faf4eaddb37f6d5da0d92660edf465f4: Status 404 returned error can't find the container with id 682bf84481e5a8e50b5a937ae9ebec44faf4eaddb37f6d5da0d92660edf465f4 Oct 31 01:31:26.515780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:26.515744 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" event={"ID":"31f02d62-d6ed-40bb-9e8b-d172d5a7d522","Type":"ContainerStarted","Data":"7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49"} Oct 31 01:31:26.516185 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:26.515794 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" event={"ID":"31f02d62-d6ed-40bb-9e8b-d172d5a7d522","Type":"ContainerStarted","Data":"682bf84481e5a8e50b5a937ae9ebec44faf4eaddb37f6d5da0d92660edf465f4"} Oct 31 01:31:26.555819 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:26.555755 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" podStartSLOduration=2.555737124 podStartE2EDuration="2.555737124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:26.55381735 +0000 UTC m=+4304.251931118" watchObservedRunningTime="2026-10-31 01:31:26.555737124 +0000 UTC m=+4304.253850890" Oct 31 01:31:27.221276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:27.221238 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:27.236471 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:27.236419 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:28.920877 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:28.920840 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:30.592908 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:30.592873 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:31.032638 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:31.032541 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv"] Oct 31 01:31:31.041789 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:31.041752 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pmqlv"] Oct 31 01:31:32.286531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:32.286488 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:32.424729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:32.424694 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740a1a3b-f68f-438f-ba0b-225b5c71f798" path="/var/lib/kubelet/pods/740a1a3b-f68f-438f-ba0b-225b5c71f798/volumes" Oct 31 01:31:33.181144 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:31:33.181109 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:31:33.896123 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:33.896081 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:35.235724 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:35.235694 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:35.248592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:35.248564 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:35.492253 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:35.492167 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:35.580035 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:35.580006 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:31:37.074184 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:37.074121 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:38.607994 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:38.607963 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:40.184345 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:40.184299 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:41.809438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:41.809392 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:43.416534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:43.416478 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:45.182073 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:45.182004 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:46.685740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:46.685698 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:48.306615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:48.306576 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:49.292053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:49.292018 5978 scope.go:122] "RemoveContainer" containerID="c56529bd07f8996039168085a0f1f76e037880c1b540d382a7529c8ce0feed24" Oct 31 01:31:49.803598 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:49.803567 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:51.388464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:51.388413 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:51.841354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:51.841312 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:53.002803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:53.002764 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:54.031109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:54.031020 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr"] Oct 31 01:31:54.040667 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:54.040631 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bhgqr"] Oct 31 01:31:54.423851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:54.423803 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd502e0d-a72f-4af0-add6-23b84b2da731" path="/var/lib/kubelet/pods/fd502e0d-a72f-4af0-add6-23b84b2da731/volumes" Oct 31 01:31:54.610411 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:54.610369 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:56.299578 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:56.299540 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:57.887931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:57.887891 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:59.511941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:31:59.511901 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:01.210222 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:01.210161 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:02.971414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:02.971372 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:03.182997 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:32:03.182960 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:32:04.590385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:04.590336 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:05.028798 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:05.028718 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4"] Oct 31 01:32:05.038354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:05.038323 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29rs4"] Oct 31 01:32:06.193835 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:06.193794 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:06.423804 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:06.423767 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e18e8c-e0b4-48ef-b949-cf11571b7e30" path="/var/lib/kubelet/pods/47e18e8c-e0b4-48ef-b949-cf11571b7e30/volumes" Oct 31 01:32:07.781436 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:07.781400 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:08.857568 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:08.857531 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:09.586941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:09.586890 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:11.403214 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:11.403179 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:13.073044 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:13.072996 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:13.778788 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:13.778750 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p"] Oct 31 01:32:13.887139 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:13.887100 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:14.588801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:14.588764 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:16.246794 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:16.246756 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:17.807257 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:17.807227 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:19.400241 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:19.400198 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:19.773088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:19.772997 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:19.914129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:19.914096 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:21.007978 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:21.007876 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:22.282623 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.282583 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz"] Oct 31 01:32:22.283063 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.282835 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" podUID="b28d1687-0830-480f-8189-277879542db1" containerName="maas-api" containerID="cri-o://a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0" gracePeriod=30 Oct 31 01:32:22.398929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.398896 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:22.749151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.749124 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:32:22.844207 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.844175 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-metrics-certs\") pod \"b28d1687-0830-480f-8189-277879542db1\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " Oct 31 01:32:22.844419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.844263 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b28d1687-0830-480f-8189-277879542db1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28d1687-0830-480f-8189-277879542db1-openshift-service-ca\") pod \"b28d1687-0830-480f-8189-277879542db1\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " Oct 31 01:32:22.844419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.844295 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b28d1687-0830-480f-8189-277879542db1-kube-api-access-979lw\" (UniqueName: \"kubernetes.io/projected/b28d1687-0830-480f-8189-277879542db1-kube-api-access-979lw\") pod \"b28d1687-0830-480f-8189-277879542db1\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " Oct 31 01:32:22.844419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.844359 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-maas-api-tls\") pod \"b28d1687-0830-480f-8189-277879542db1\" (UID: \"b28d1687-0830-480f-8189-277879542db1\") " Oct 31 01:32:22.844794 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.844753 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28d1687-0830-480f-8189-277879542db1-openshift-service-ca" pod "b28d1687-0830-480f-8189-277879542db1" (UID: "b28d1687-0830-480f-8189-277879542db1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:22.846579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.846539 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-maas-api-tls" pod "b28d1687-0830-480f-8189-277879542db1" (UID: "b28d1687-0830-480f-8189-277879542db1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:22.846874 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.846850 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28d1687-0830-480f-8189-277879542db1-kube-api-access-979lw" pod "b28d1687-0830-480f-8189-277879542db1" (UID: "b28d1687-0830-480f-8189-277879542db1"). InnerVolumeSpecName "kube-api-access-979lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:22.847364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.847341 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-metrics-certs" pod "b28d1687-0830-480f-8189-277879542db1" (UID: "b28d1687-0830-480f-8189-277879542db1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:22.850158 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.850135 5978 generic.go:356] "Generic (PLEG): container finished" podID="b28d1687-0830-480f-8189-277879542db1" containerID="a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0" exitCode=0 Oct 31 01:32:22.850257 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.850169 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" event={"ID":"b28d1687-0830-480f-8189-277879542db1","Type":"ContainerDied","Data":"a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0"} Oct 31 01:32:22.850257 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.850182 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" Oct 31 01:32:22.850257 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.850211 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz" event={"ID":"b28d1687-0830-480f-8189-277879542db1","Type":"ContainerDied","Data":"94be194da30eddfbd64691ece0278a8142157f718530f460313df3b973a64432"} Oct 31 01:32:22.850257 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.850232 5978 scope.go:122] "RemoveContainer" containerID="a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0" Oct 31 01:32:22.868959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.868939 5978 scope.go:122] "RemoveContainer" containerID="a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0" Oct 31 01:32:22.869288 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:32:22.869266 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0\": container with ID starting with a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0 not found: ID does not exist" containerID="a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0" Oct 31 01:32:22.869390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.869297 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0"} err="failed to get container status \"a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0\": rpc error: code = NotFound desc = could not find container \"a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0\": container with ID starting with a5c93a725273fb17a6a6fbcc091e4ff52b61b4890e30e811fc3d7d90971a2df0 not found: ID does not exist" Oct 31 01:32:22.874335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.874307 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:22.899916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.899882 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz"] Oct 31 01:32:22.911198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.911165 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e8e612-8964fb75-7t9xz"] Oct 31 01:32:22.945776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.945740 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:22.945776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.945770 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28d1687-0830-480f-8189-277879542db1-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:22.945776 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.945781 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-979lw\" (UniqueName: \"kubernetes.io/projected/b28d1687-0830-480f-8189-277879542db1-kube-api-access-979lw\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:22.946027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:22.945790 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b28d1687-0830-480f-8189-277879542db1-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:23.033683 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:23.033651 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:23.578262 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:23.578225 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:24.426742 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:24.426707 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b28d1687-0830-480f-8189-277879542db1" path="/var/lib/kubelet/pods/b28d1687-0830-480f-8189-277879542db1/volumes" Oct 31 01:32:24.933446 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:24.933406 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:28.801195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:28.801150 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" podUID="31f02d62-d6ed-40bb-9e8b-d172d5a7d522" containerName="main" containerID="cri-o://7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49" gracePeriod=45 Oct 31 01:32:28.860315 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:32:28.860280 5978 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31f02d62_d6ed_40bb_9e8b_d172d5a7d522.slice/crio-conmon-7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:32:29.059131 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.059072 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:32:29.107995 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.107964 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tls-certs\" (UniqueName: \"kubernetes.io/secret/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tls-certs\") pod \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " Oct 31 01:32:29.108441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108088 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-dshm\") pod \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " Oct 31 01:32:29.108441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108119 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-home\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-home\") pod \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " Oct 31 01:32:29.108441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108154 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-model-cache\") pod \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " Oct 31 01:32:29.108441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108327 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-kube-api-access-zcx6v\" (UniqueName: \"kubernetes.io/projected/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-kube-api-access-zcx6v\") pod \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " Oct 31 01:32:29.108441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108415 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tmp-dir\") pod \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\" (UID: \"31f02d62-d6ed-40bb-9e8b-d172d5a7d522\") " Oct 31 01:32:29.108802 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108490 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-home" pod "31f02d62-d6ed-40bb-9e8b-d172d5a7d522" (UID: "31f02d62-d6ed-40bb-9e8b-d172d5a7d522"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:29.108802 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108649 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-model-cache" pod "31f02d62-d6ed-40bb-9e8b-d172d5a7d522" (UID: "31f02d62-d6ed-40bb-9e8b-d172d5a7d522"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:29.108928 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108840 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tmp-dir" pod "31f02d62-d6ed-40bb-9e8b-d172d5a7d522" (UID: "31f02d62-d6ed-40bb-9e8b-d172d5a7d522"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:29.108985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108931 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:29.108985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108949 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:29.108985 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.108964 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:29.110768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.110741 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tls-certs" pod "31f02d62-d6ed-40bb-9e8b-d172d5a7d522" (UID: "31f02d62-d6ed-40bb-9e8b-d172d5a7d522"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:29.110914 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.110844 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-dshm" pod "31f02d62-d6ed-40bb-9e8b-d172d5a7d522" (UID: "31f02d62-d6ed-40bb-9e8b-d172d5a7d522"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:29.111784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.111760 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-kube-api-access-zcx6v" pod "31f02d62-d6ed-40bb-9e8b-d172d5a7d522" (UID: "31f02d62-d6ed-40bb-9e8b-d172d5a7d522"). InnerVolumeSpecName "kube-api-access-zcx6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:29.210631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.210591 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:29.210631 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.210628 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcx6v\" (UniqueName: \"kubernetes.io/projected/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-kube-api-access-zcx6v\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:29.210910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.210642 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f02d62-d6ed-40bb-9e8b-d172d5a7d522-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:29.457801 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.457761 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:29.600608 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.600555 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:29.838276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.838244 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:29.898007 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.897969 5978 generic.go:356] "Generic (PLEG): container finished" podID="31f02d62-d6ed-40bb-9e8b-d172d5a7d522" containerID="7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49" exitCode=0 Oct 31 01:32:29.898203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.898017 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" Oct 31 01:32:29.898203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.898056 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" event={"ID":"31f02d62-d6ed-40bb-9e8b-d172d5a7d522","Type":"ContainerDied","Data":"7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49"} Oct 31 01:32:29.898203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.898105 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p" event={"ID":"31f02d62-d6ed-40bb-9e8b-d172d5a7d522","Type":"ContainerDied","Data":"682bf84481e5a8e50b5a937ae9ebec44faf4eaddb37f6d5da0d92660edf465f4"} Oct 31 01:32:29.898203 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.898128 5978 scope.go:122] "RemoveContainer" containerID="7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49" Oct 31 01:32:29.909082 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.909056 5978 scope.go:122] "RemoveContainer" containerID="7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49" Oct 31 01:32:29.909402 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:32:29.909381 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49\": container with ID starting with 7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49 not found: ID does not exist" containerID="7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49" Oct 31 01:32:29.909485 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.909412 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49"} err="failed to get container status \"7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49\": rpc error: code = NotFound desc = could not find container \"7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49\": container with ID starting with 7ee9622db05332a013304da7c4e12ced163af52876f9adcebf7a9ea45b444c49 not found: ID does not exist" Oct 31 01:32:29.944168 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.944119 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p"] Oct 31 01:32:29.953606 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:29.953575 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-e8e612/simulator-single-main-e8e612-kserve-8cdfb6b99-mpk9p"] Oct 31 01:32:30.424280 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:30.424244 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f02d62-d6ed-40bb-9e8b-d172d5a7d522" path="/var/lib/kubelet/pods/31f02d62-d6ed-40bb-9e8b-d172d5a7d522/volumes" Oct 31 01:32:31.064246 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.064213 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf"] Oct 31 01:32:31.073547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.073519 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fgmf"] Oct 31 01:32:31.112368 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.112332 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2"] Oct 31 01:32:31.113003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.112980 5978 cpu_manager.go:414] "containerMap: removing container" podUID="31f02d62-d6ed-40bb-9e8b-d172d5a7d522" containerName="main" Oct 31 01:32:31.113080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.113006 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f02d62-d6ed-40bb-9e8b-d172d5a7d522" containerName="main" Oct 31 01:32:31.113080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.113030 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b28d1687-0830-480f-8189-277879542db1" containerName="maas-api" Oct 31 01:32:31.113080 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.113040 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b28d1687-0830-480f-8189-277879542db1" containerName="maas-api" Oct 31 01:32:31.113181 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.113142 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f02d62-d6ed-40bb-9e8b-d172d5a7d522" containerName="main" Oct 31 01:32:31.113181 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.113151 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b28d1687-0830-480f-8189-277879542db1" containerName="maas-api" Oct 31 01:32:31.117664 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.117649 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.129912 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.129886 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:32:31.130086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.130066 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:32:31.130142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.130085 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:32:31.130142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.130133 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:32:31.136392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.136363 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2"] Oct 31 01:32:31.232040 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.232008 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nb8d\" (UniqueName: \"kubernetes.io/projected/b962734d-5d45-49fa-8c0c-2037e11c08da-kube-api-access-4nb8d\") pod \"maas-api-revoke-sub-3d4156f1-7n7c2\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.232248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.232152 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7n7c2\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.232248 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.232190 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7n7c2\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.333848 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.333810 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7n7c2\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.334022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.333863 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7n7c2\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.334022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.333950 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nb8d\" (UniqueName: \"kubernetes.io/projected/b962734d-5d45-49fa-8c0c-2037e11c08da-kube-api-access-4nb8d\") pod \"maas-api-revoke-sub-3d4156f1-7n7c2\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.334507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.334483 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7n7c2\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.336129 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.336104 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7n7c2\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.354943 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.354914 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nb8d\" (UniqueName: \"kubernetes.io/projected/b962734d-5d45-49fa-8c0c-2037e11c08da-kube-api-access-4nb8d\") pod \"maas-api-revoke-sub-3d4156f1-7n7c2\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.430956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.430919 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:31.726159 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.726128 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2"] Oct 31 01:32:31.728172 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:32:31.728141 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb962734d_5d45_49fa_8c0c_2037e11c08da.slice/crio-a01fa4cf656ea64d783ca5af6a6cc3f65a5a21dcbd564296e465f22c28059f57 WatchSource:0}: Error finding container a01fa4cf656ea64d783ca5af6a6cc3f65a5a21dcbd564296e465f22c28059f57: Status 404 returned error can't find the container with id a01fa4cf656ea64d783ca5af6a6cc3f65a5a21dcbd564296e465f22c28059f57 Oct 31 01:32:31.912959 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:31.912921 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" event={"ID":"b962734d-5d45-49fa-8c0c-2037e11c08da","Type":"ContainerStarted","Data":"a01fa4cf656ea64d783ca5af6a6cc3f65a5a21dcbd564296e465f22c28059f57"} Oct 31 01:32:32.424441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:32.424406 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981d5ad5-2fc1-4a43-bd89-20f1711d5e6f" path="/var/lib/kubelet/pods/981d5ad5-2fc1-4a43-bd89-20f1711d5e6f/volumes" Oct 31 01:32:32.920402 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:32.920370 5978 generic.go:356] "Generic (PLEG): container finished" podID="b962734d-5d45-49fa-8c0c-2037e11c08da" containerID="eab980c99e1257b49feb2040e2503cc70b63d00f8a1130921870ac3ea9064d0c" exitCode=0 Oct 31 01:32:32.920599 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:32.920412 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" event={"ID":"b962734d-5d45-49fa-8c0c-2037e11c08da","Type":"ContainerDied","Data":"eab980c99e1257b49feb2040e2503cc70b63d00f8a1130921870ac3ea9064d0c"} Oct 31 01:32:33.027882 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:33.027847 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq"] Oct 31 01:32:33.030489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:33.030444 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-25skq"] Oct 31 01:32:33.181127 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:32:33.181047 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:32:34.063387 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.063362 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:34.172549 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.172513 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b962734d-5d45-49fa-8c0c-2037e11c08da-kube-api-access-4nb8d\" (UniqueName: \"kubernetes.io/projected/b962734d-5d45-49fa-8c0c-2037e11c08da-kube-api-access-4nb8d\") pod \"b962734d-5d45-49fa-8c0c-2037e11c08da\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " Oct 31 01:32:34.172728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.172625 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-xks-ca\") pod \"b962734d-5d45-49fa-8c0c-2037e11c08da\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " Oct 31 01:32:34.172728 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.172695 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-openshift-ca\") pod \"b962734d-5d45-49fa-8c0c-2037e11c08da\" (UID: \"b962734d-5d45-49fa-8c0c-2037e11c08da\") " Oct 31 01:32:34.173051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.173030 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-openshift-ca" pod "b962734d-5d45-49fa-8c0c-2037e11c08da" (UID: "b962734d-5d45-49fa-8c0c-2037e11c08da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:34.174731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.174701 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-xks-ca" pod "b962734d-5d45-49fa-8c0c-2037e11c08da" (UID: "b962734d-5d45-49fa-8c0c-2037e11c08da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:34.174858 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.174768 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b962734d-5d45-49fa-8c0c-2037e11c08da-kube-api-access-4nb8d" pod "b962734d-5d45-49fa-8c0c-2037e11c08da" (UID: "b962734d-5d45-49fa-8c0c-2037e11c08da"). InnerVolumeSpecName "kube-api-access-4nb8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:34.273576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.273543 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nb8d\" (UniqueName: \"kubernetes.io/projected/b962734d-5d45-49fa-8c0c-2037e11c08da-kube-api-access-4nb8d\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:34.273576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.273573 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:34.273576 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.273582 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962734d-5d45-49fa-8c0c-2037e11c08da-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:32:34.424425 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.424389 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77be76af-a574-49c2-8979-32e92e31c33a" path="/var/lib/kubelet/pods/77be76af-a574-49c2-8979-32e92e31c33a/volumes" Oct 31 01:32:34.933236 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.933210 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" Oct 31 01:32:34.933488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.933237 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2" event={"ID":"b962734d-5d45-49fa-8c0c-2037e11c08da","Type":"ContainerDied","Data":"a01fa4cf656ea64d783ca5af6a6cc3f65a5a21dcbd564296e465f22c28059f57"} Oct 31 01:32:34.933488 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:34.933294 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a01fa4cf656ea64d783ca5af6a6cc3f65a5a21dcbd564296e465f22c28059f57" Oct 31 01:32:44.077675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:44.077636 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb"] Oct 31 01:32:44.109844 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:44.109803 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r45wb"] Oct 31 01:32:44.450528 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:44.449999 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a2878f-0abf-4098-b8a1-6ef92cd23591" path="/var/lib/kubelet/pods/20a2878f-0abf-4098-b8a1-6ef92cd23591/volumes" Oct 31 01:32:45.623633 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:45.623585 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ldf84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:32:45.624290 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:45.624253 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ldf84" podUID="95964f90-7a0d-4fe8-97aa-6f10cf619a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:32:45.624952 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:45.624926 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6bxb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:32:45.625145 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:45.625121 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6bxb6" podUID="694e2b84-5e34-4d1e-bf99-16b261a95272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:32:45.755731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:45.755686 5978 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qq9v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:32:45.755944 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:45.755758 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qq9v6" podUID="b4643839-1a91-40bd-b0f2-d335540a1ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:32:48.189215 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:48.189175 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:49.317438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:49.317407 5978 scope.go:122] "RemoveContainer" containerID="769c79c0fc7bf6e1bc3e73bbfa28966a29a161a1867d0aa077f0d309836ef895" Oct 31 01:32:50.000562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:50.000515 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:51.480597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:51.480560 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:51.968694 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:51.968658 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:57.680851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:32:57.680814 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:33:03.181710 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:33:03.181669 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:33:06.041394 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:06.041356 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6"] Oct 31 01:33:06.050800 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:06.050766 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lljj6"] Oct 31 01:33:06.424485 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:06.424435 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8224548f-686b-41cd-b027-98b0daa85c2f" path="/var/lib/kubelet/pods/8224548f-686b-41cd-b027-98b0daa85c2f/volumes" Oct 31 01:33:28.031688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:28.031593 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95"] Oct 31 01:33:28.041186 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:28.041154 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-q4k95"] Oct 31 01:33:28.423893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:28.423858 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950786e0-b57f-4452-8028-2cdd94b6cad7" path="/var/lib/kubelet/pods/950786e0-b57f-4452-8028-2cdd94b6cad7/volumes" Oct 31 01:33:33.181154 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:33:33.181113 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:33:49.030017 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:49.029983 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9"] Oct 31 01:33:49.039538 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:49.039510 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-chxk9"] Oct 31 01:33:49.351137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:49.351103 5978 scope.go:122] "RemoveContainer" containerID="ddd715c2420a5f7314da6a8b752a651081a0fa7d20c0f5c4192aee1d3623f91a" Oct 31 01:33:49.360409 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:49.360386 5978 scope.go:122] "RemoveContainer" containerID="bf7f62cf77d6b02bb754ef7c482664366d36f6f78d6b65a1409d2461d09868d6" Oct 31 01:33:49.368960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:49.368938 5978 scope.go:122] "RemoveContainer" containerID="4d9831a016fb55c40571ebe7a0b52fe68a01d55937f273ce05424bea5f53bcaa" Oct 31 01:33:49.377579 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:49.377559 5978 scope.go:122] "RemoveContainer" containerID="a8ce3cf5a028dc4e648d940b0072ae0729d6c0298827a4b6388f857967060dcb" Oct 31 01:33:50.424226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:50.424193 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329d88be-8e2c-4b9c-b74b-474f12957f5a" path="/var/lib/kubelet/pods/329d88be-8e2c-4b9c-b74b-474f12957f5a/volumes" Oct 31 01:33:52.099574 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:52.099534 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:33:52.251008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:33:52.250966 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:03.181829 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:34:03.181786 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:34:09.041962 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:09.041920 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn"] Oct 31 01:34:09.051388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:09.051359 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vhwnn"] Oct 31 01:34:10.423534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:10.423496 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c489ad-d658-4ff8-90ee-ea39aceede02" path="/var/lib/kubelet/pods/e5c489ad-d658-4ff8-90ee-ea39aceede02/volumes" Oct 31 01:34:20.575059 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.575021 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775"] Oct 31 01:34:20.575582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.575561 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b962734d-5d45-49fa-8c0c-2037e11c08da" containerName="revoke-keys" Oct 31 01:34:20.575669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.575584 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b962734d-5d45-49fa-8c0c-2037e11c08da" containerName="revoke-keys" Oct 31 01:34:20.575725 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.575711 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b962734d-5d45-49fa-8c0c-2037e11c08da" containerName="revoke-keys" Oct 31 01:34:20.578981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.578961 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.602048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.601928 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:34:20.602048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.601949 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:34:20.602048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.602011 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:34:20.602048 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.602027 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:34:20.604255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.604226 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775"] Oct 31 01:34:20.730785 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.730746 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cs775\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.730785 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.730787 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cs775\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.731050 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.730918 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn9hp\" (UniqueName: \"kubernetes.io/projected/e3893383-addc-4ec8-9d4d-0ccf412b1857-kube-api-access-vn9hp\") pod \"maas-api-revoke-sub-d576e68b-cs775\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.832116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.832020 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cs775\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.832116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.832064 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cs775\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.832116 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.832111 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn9hp\" (UniqueName: \"kubernetes.io/projected/e3893383-addc-4ec8-9d4d-0ccf412b1857-kube-api-access-vn9hp\") pod \"maas-api-revoke-sub-d576e68b-cs775\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.832811 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.832783 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cs775\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.834448 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.834420 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cs775\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.853389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.853364 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn9hp\" (UniqueName: \"kubernetes.io/projected/e3893383-addc-4ec8-9d4d-0ccf412b1857-kube-api-access-vn9hp\") pod \"maas-api-revoke-sub-d576e68b-cs775\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:20.892502 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:20.892471 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:21.189161 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:21.189132 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775"] Oct 31 01:34:21.195366 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:34:21.195313 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3893383_addc_4ec8_9d4d_0ccf412b1857.slice/crio-185ca84ff39b2d5c0ac829bb95e6b8f43723cc5117e08fffcd8e70b9645de034 WatchSource:0}: Error finding container 185ca84ff39b2d5c0ac829bb95e6b8f43723cc5117e08fffcd8e70b9645de034: Status 404 returned error can't find the container with id 185ca84ff39b2d5c0ac829bb95e6b8f43723cc5117e08fffcd8e70b9645de034 Oct 31 01:34:21.650000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:21.649959 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" event={"ID":"e3893383-addc-4ec8-9d4d-0ccf412b1857","Type":"ContainerStarted","Data":"26b4d3a05e8bfe0614f66a6fc62c452e48f31a9cc3dba4ad29118e567eb3f11b"} Oct 31 01:34:21.650517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:21.650023 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" event={"ID":"e3893383-addc-4ec8-9d4d-0ccf412b1857","Type":"ContainerStarted","Data":"185ca84ff39b2d5c0ac829bb95e6b8f43723cc5117e08fffcd8e70b9645de034"} Oct 31 01:34:21.688346 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:21.688293 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" podStartSLOduration=1.6882766820000001 podStartE2EDuration="1.688276682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:21.68640124 +0000 UTC m=+4479.384515019" watchObservedRunningTime="2026-10-31 01:34:21.688276682 +0000 UTC m=+4479.386390449" Oct 31 01:34:22.657867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:22.657829 5978 generic.go:356] "Generic (PLEG): container finished" podID="e3893383-addc-4ec8-9d4d-0ccf412b1857" containerID="26b4d3a05e8bfe0614f66a6fc62c452e48f31a9cc3dba4ad29118e567eb3f11b" exitCode=0 Oct 31 01:34:22.658295 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:22.657910 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" event={"ID":"e3893383-addc-4ec8-9d4d-0ccf412b1857","Type":"ContainerDied","Data":"26b4d3a05e8bfe0614f66a6fc62c452e48f31a9cc3dba4ad29118e567eb3f11b"} Oct 31 01:34:23.803806 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:23.803783 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:23.965618 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:23.965520 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3893383-addc-4ec8-9d4d-0ccf412b1857-kube-api-access-vn9hp\" (UniqueName: \"kubernetes.io/projected/e3893383-addc-4ec8-9d4d-0ccf412b1857-kube-api-access-vn9hp\") pod \"e3893383-addc-4ec8-9d4d-0ccf412b1857\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " Oct 31 01:34:23.965810 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:23.965657 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-openshift-ca\") pod \"e3893383-addc-4ec8-9d4d-0ccf412b1857\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " Oct 31 01:34:23.965810 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:23.965741 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-xks-ca\") pod \"e3893383-addc-4ec8-9d4d-0ccf412b1857\" (UID: \"e3893383-addc-4ec8-9d4d-0ccf412b1857\") " Oct 31 01:34:23.966110 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:23.966082 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-openshift-ca" pod "e3893383-addc-4ec8-9d4d-0ccf412b1857" (UID: "e3893383-addc-4ec8-9d4d-0ccf412b1857"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:34:23.967665 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:23.967640 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3893383-addc-4ec8-9d4d-0ccf412b1857-kube-api-access-vn9hp" pod "e3893383-addc-4ec8-9d4d-0ccf412b1857" (UID: "e3893383-addc-4ec8-9d4d-0ccf412b1857"). InnerVolumeSpecName "kube-api-access-vn9hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:34:23.967784 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:23.967683 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-xks-ca" pod "e3893383-addc-4ec8-9d4d-0ccf412b1857" (UID: "e3893383-addc-4ec8-9d4d-0ccf412b1857"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:34:24.066828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:24.066789 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:34:24.066828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:24.066821 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3893383-addc-4ec8-9d4d-0ccf412b1857-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:34:24.066828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:24.066831 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn9hp\" (UniqueName: \"kubernetes.io/projected/e3893383-addc-4ec8-9d4d-0ccf412b1857-kube-api-access-vn9hp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:34:24.671428 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:24.671392 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" Oct 31 01:34:24.671639 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:24.671468 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775" event={"ID":"e3893383-addc-4ec8-9d4d-0ccf412b1857","Type":"ContainerDied","Data":"185ca84ff39b2d5c0ac829bb95e6b8f43723cc5117e08fffcd8e70b9645de034"} Oct 31 01:34:24.671639 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:24.671514 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="185ca84ff39b2d5c0ac829bb95e6b8f43723cc5117e08fffcd8e70b9645de034" Oct 31 01:34:33.181105 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:34:33.181066 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:34:42.126107 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.126070 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk"] Oct 31 01:34:42.126709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.126517 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e3893383-addc-4ec8-9d4d-0ccf412b1857" containerName="revoke-keys" Oct 31 01:34:42.126709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.126530 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3893383-addc-4ec8-9d4d-0ccf412b1857" containerName="revoke-keys" Oct 31 01:34:42.126709 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.126673 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3893383-addc-4ec8-9d4d-0ccf412b1857" containerName="revoke-keys" Oct 31 01:34:42.130149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.130129 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.146597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.146556 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:34:42.154867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.154846 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:34:42.154969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.154911 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:34:42.154969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.154926 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:34:42.163723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.163695 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk"] Oct 31 01:34:42.225782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.225749 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-m6qvk\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.225974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.225852 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-m6qvk\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.225974 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.225901 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtr5l\" (UniqueName: \"kubernetes.io/projected/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-kube-api-access-wtr5l\") pod \"maas-api-revoke-sub-d2bc93bc-m6qvk\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.326960 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.326919 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-m6qvk\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.327156 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.326976 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-m6qvk\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.327156 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.326997 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtr5l\" (UniqueName: \"kubernetes.io/projected/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-kube-api-access-wtr5l\") pod \"maas-api-revoke-sub-d2bc93bc-m6qvk\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.327770 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.327741 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-m6qvk\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.329331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.329310 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-m6qvk\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.349507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.349448 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtr5l\" (UniqueName: \"kubernetes.io/projected/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-kube-api-access-wtr5l\") pod \"maas-api-revoke-sub-d2bc93bc-m6qvk\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.443930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.443845 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:42.747537 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.747505 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk"] Oct 31 01:34:42.750592 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:34:42.750550 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd583e6a7_7b53_40a0_bc77_cc6e72cf613a.slice/crio-1cadb47c0869988631c661fa1a5ce007fddd00e848af8eb949af2fee4d5a59ff WatchSource:0}: Error finding container 1cadb47c0869988631c661fa1a5ce007fddd00e848af8eb949af2fee4d5a59ff: Status 404 returned error can't find the container with id 1cadb47c0869988631c661fa1a5ce007fddd00e848af8eb949af2fee4d5a59ff Oct 31 01:34:42.792838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:42.792808 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" event={"ID":"d583e6a7-7b53-40a0-bc77-cc6e72cf613a","Type":"ContainerStarted","Data":"1cadb47c0869988631c661fa1a5ce007fddd00e848af8eb949af2fee4d5a59ff"} Oct 31 01:34:43.800934 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:43.800891 5978 generic.go:356] "Generic (PLEG): container finished" podID="d583e6a7-7b53-40a0-bc77-cc6e72cf613a" containerID="693a6d58e7ea042ad9473e241786d71d180428f1fb0b96aedf82d1cb076986b9" exitCode=0 Oct 31 01:34:43.801447 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:43.800936 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" event={"ID":"d583e6a7-7b53-40a0-bc77-cc6e72cf613a","Type":"ContainerDied","Data":"693a6d58e7ea042ad9473e241786d71d180428f1fb0b96aedf82d1cb076986b9"} Oct 31 01:34:44.942998 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:44.942970 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:45.053799 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.053762 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-kube-api-access-wtr5l\" (UniqueName: \"kubernetes.io/projected/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-kube-api-access-wtr5l\") pod \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " Oct 31 01:34:45.054007 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.053810 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-openshift-ca\") pod \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " Oct 31 01:34:45.054007 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.053842 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-xks-ca\") pod \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\" (UID: \"d583e6a7-7b53-40a0-bc77-cc6e72cf613a\") " Oct 31 01:34:45.054254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.054230 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-openshift-ca" pod "d583e6a7-7b53-40a0-bc77-cc6e72cf613a" (UID: "d583e6a7-7b53-40a0-bc77-cc6e72cf613a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:34:45.055950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.055920 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-xks-ca" pod "d583e6a7-7b53-40a0-bc77-cc6e72cf613a" (UID: "d583e6a7-7b53-40a0-bc77-cc6e72cf613a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:34:45.056053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.055971 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-kube-api-access-wtr5l" pod "d583e6a7-7b53-40a0-bc77-cc6e72cf613a" (UID: "d583e6a7-7b53-40a0-bc77-cc6e72cf613a"). InnerVolumeSpecName "kube-api-access-wtr5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:34:45.155255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.155217 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtr5l\" (UniqueName: \"kubernetes.io/projected/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-kube-api-access-wtr5l\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:34:45.155255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.155247 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:34:45.155255 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.155260 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d583e6a7-7b53-40a0-bc77-cc6e72cf613a-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:34:45.812992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.812960 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" Oct 31 01:34:45.813180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.812992 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk" event={"ID":"d583e6a7-7b53-40a0-bc77-cc6e72cf613a","Type":"ContainerDied","Data":"1cadb47c0869988631c661fa1a5ce007fddd00e848af8eb949af2fee4d5a59ff"} Oct 31 01:34:45.813180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:45.813040 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cadb47c0869988631c661fa1a5ce007fddd00e848af8eb949af2fee4d5a59ff" Oct 31 01:34:49.415063 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:49.415024 5978 scope.go:122] "RemoveContainer" containerID="0d883e1f30132074d256b6a6134682c7678fe613481578ba3d30b6d8fa78f20f" Oct 31 01:34:49.424605 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:49.424584 5978 scope.go:122] "RemoveContainer" containerID="71a333ca7a8441954dde90156d01d179ae0354a6926588d013d8fd19045ad3de" Oct 31 01:34:49.436699 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:49.436676 5978 scope.go:122] "RemoveContainer" containerID="25c6b90973bf35a6840da05ae0ea4bcbaedcffd02b1c83674d2bbb048a7b4403" Oct 31 01:34:50.770556 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:50.770521 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:34:50.780852 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:50.780828 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:34:52.243300 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:52.243261 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:53.918118 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:53.918028 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:54.500210 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:54.500169 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:54.671430 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:54.671397 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:57.875386 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:57.875350 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:58.056526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:58.056484 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:59.234369 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.234328 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8"] Oct 31 01:34:59.235133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.235108 5978 cpu_manager.go:414] "containerMap: removing container" podUID="d583e6a7-7b53-40a0-bc77-cc6e72cf613a" containerName="revoke-keys" Oct 31 01:34:59.235133 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.235133 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d583e6a7-7b53-40a0-bc77-cc6e72cf613a" containerName="revoke-keys" Oct 31 01:34:59.235307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.235236 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="d583e6a7-7b53-40a0-bc77-cc6e72cf613a" containerName="revoke-keys" Oct 31 01:34:59.238346 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.238327 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.253534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.253506 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:34:59.253534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.253534 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:34:59.263832 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.263563 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2e17bf-serving-cert\"" Oct 31 01:34:59.263832 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.263649 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cckh7\"" Oct 31 01:34:59.264438 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.264029 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:34:59.267742 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.267716 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8"] Oct 31 01:34:59.301912 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.301866 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6k8j\" (UniqueName: \"kubernetes.io/projected/bac09980-4503-41db-acf7-3e24a2a44c46-kube-api-access-p6k8j\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.302102 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.302039 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.302157 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.302137 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac09980-4503-41db-acf7-3e24a2a44c46-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.302235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.302221 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.365215 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.365167 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:59.403770 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.403741 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.403929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.403805 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6k8j\" (UniqueName: \"kubernetes.io/projected/bac09980-4503-41db-acf7-3e24a2a44c46-kube-api-access-p6k8j\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.403929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.403846 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.403929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.403875 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac09980-4503-41db-acf7-3e24a2a44c46-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.404506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.404483 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac09980-4503-41db-acf7-3e24a2a44c46-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.406266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.406244 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.406370 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.406322 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.429659 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.429627 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6k8j\" (UniqueName: \"kubernetes.io/projected/bac09980-4503-41db-acf7-3e24a2a44c46-kube-api-access-p6k8j\") pod \"maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.553214 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.553130 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:59.559180 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.559160 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:34:59.729938 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.729902 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:59.872051 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.872021 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8"] Oct 31 01:34:59.874812 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:34:59.874785 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac09980_4503_41db_acf7_3e24a2a44c46.slice/crio-73627b87345f67df0c37267af1f5957319f8d24827c9a68213361890dac50fb5 WatchSource:0}: Error finding container 73627b87345f67df0c37267af1f5957319f8d24827c9a68213361890dac50fb5: Status 404 returned error can't find the container with id 73627b87345f67df0c37267af1f5957319f8d24827c9a68213361890dac50fb5 Oct 31 01:34:59.894723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:34:59.894689 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" event={"ID":"bac09980-4503-41db-acf7-3e24a2a44c46","Type":"ContainerStarted","Data":"73627b87345f67df0c37267af1f5957319f8d24827c9a68213361890dac50fb5"} Oct 31 01:35:00.902359 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:00.902321 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" event={"ID":"bac09980-4503-41db-acf7-3e24a2a44c46","Type":"ContainerStarted","Data":"6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9"} Oct 31 01:35:00.902835 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:00.902480 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:35:00.976986 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:00.976926 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" podStartSLOduration=1.613680724 podStartE2EDuration="1.976908114s" totalImagesPullingTime="363.22739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:00.974993014 +0000 UTC m=+4518.673106778" watchObservedRunningTime="2026-10-31 01:35:00.976908114 +0000 UTC m=+4518.675021878" Oct 31 01:35:01.227593 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:01.227480 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:02.208803 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:02.208758 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:02.359039 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:02.359002 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:02.890868 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:02.890828 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:03.181398 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:35:03.181312 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:35:04.510013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:04.509973 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:06.290383 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:06.290337 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:06.914622 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:06.914592 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:35:08.067361 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:08.067324 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:09.929937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:09.929897 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:11.498768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:11.498725 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:11.641615 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:11.641581 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:12.748307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:12.748267 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:13.129536 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:13.129495 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:13.562277 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:13.562182 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:13.763883 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:13.763849 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:15.245155 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:15.245106 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:17.004981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.004944 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:17.316649 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.316617 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph"] Oct 31 01:35:17.322331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.322302 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.350508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.350477 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2e17bf\"/\"openshift-service-ca.crt\"" Oct 31 01:35:17.350897 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.350597 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2e17bf\"/\"kube-root-ca.crt\"" Oct 31 01:35:17.351522 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.351502 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2e17bf\"/\"default-dockercfg-r5qvr\"" Oct 31 01:35:17.351630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.351598 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2e17bf\"/\"simulator-single-main-2e17bf-kserve-self-signed-certs\"" Oct 31 01:35:17.352763 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.352741 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph"] Oct 31 01:35:17.405414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.405373 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-tmp-dir\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.405414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.405419 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-home\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.405688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.405566 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-model-cache\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.405688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.405600 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a38c67-6349-4082-bd86-80e1a2d6724a-tls-certs\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.405688 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.405666 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-dshm\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.405851 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.405724 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m6zk\" (UniqueName: \"kubernetes.io/projected/00a38c67-6349-4082-bd86-80e1a2d6724a-kube-api-access-7m6zk\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.424468 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.424419 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:17.506427 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.506395 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6zk\" (UniqueName: \"kubernetes.io/projected/00a38c67-6349-4082-bd86-80e1a2d6724a-kube-api-access-7m6zk\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.506685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.506447 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-tmp-dir\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.506685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.506516 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-home\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.506685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.506575 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-model-cache\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.506931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.506709 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a38c67-6349-4082-bd86-80e1a2d6724a-tls-certs\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.506931 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.506775 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-dshm\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.507045 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.506990 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-tmp-dir\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.507045 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.507013 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-model-cache\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.507146 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.507081 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-home\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.508969 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.508941 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-dshm\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.509222 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.509203 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a38c67-6349-4082-bd86-80e1a2d6724a-tls-certs\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.529529 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.529499 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m6zk\" (UniqueName: \"kubernetes.io/projected/00a38c67-6349-4082-bd86-80e1a2d6724a-kube-api-access-7m6zk\") pod \"simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.639320 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.639218 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:17.940042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:17.940008 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph"] Oct 31 01:35:17.942346 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:35:17.942315 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00a38c67_6349_4082_bd86_80e1a2d6724a.slice/crio-75c27a9e98390a1748602614420c2d7993cb811f2734977cf9e7659c58fa42af WatchSource:0}: Error finding container 75c27a9e98390a1748602614420c2d7993cb811f2734977cf9e7659c58fa42af: Status 404 returned error can't find the container with id 75c27a9e98390a1748602614420c2d7993cb811f2734977cf9e7659c58fa42af Oct 31 01:35:18.002033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:18.002003 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" event={"ID":"00a38c67-6349-4082-bd86-80e1a2d6724a","Type":"ContainerStarted","Data":"047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495"} Oct 31 01:35:18.002161 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:18.002047 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" event={"ID":"00a38c67-6349-4082-bd86-80e1a2d6724a","Type":"ContainerStarted","Data":"75c27a9e98390a1748602614420c2d7993cb811f2734977cf9e7659c58fa42af"} Oct 31 01:35:18.041550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:18.041487 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" podStartSLOduration=1.041441172 podStartE2EDuration="1.041441172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:18.040248452 +0000 UTC m=+4535.738362218" watchObservedRunningTime="2026-10-31 01:35:18.041441172 +0000 UTC m=+4535.739554986" Oct 31 01:35:18.639384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:18.639350 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:18.722432 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:18.722394 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:20.586264 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:20.586227 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:22.414042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:22.414004 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:24.193541 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:24.193495 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:26.038792 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:26.038752 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr"] Oct 31 01:35:26.047891 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:26.047860 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nchcr"] Oct 31 01:35:26.087124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:26.087081 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:26.424256 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:26.424217 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaba58e4-a2fd-4fe1-94d2-666fa52ae283" path="/var/lib/kubelet/pods/eaba58e4-a2fd-4fe1-94d2-666fa52ae283/volumes" Oct 31 01:35:27.639804 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:27.639763 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:27.652274 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:27.652246 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:27.781644 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:27.781613 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:28.073580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:28.073547 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:35:29.510331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:29.510292 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:31.213710 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:31.213670 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:32.805176 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:32.805129 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:33.182551 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:35:33.182514 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:35:34.752477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:34.749059 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:36.324843 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:36.324797 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:38.013622 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:38.013573 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:39.641573 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:39.641534 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:41.393252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:41.393216 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:42.988278 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:42.988233 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:44.680211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:44.680164 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:46.299573 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:46.299543 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:47.936024 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:47.935989 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:49.466148 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:49.466112 5978 scope.go:122] "RemoveContainer" containerID="6d1ee3cd32f106cfe3dffc75c4b7cdd196eebbff5fbef9615e3aa4681bec6639" Oct 31 01:35:49.475436 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:49.475409 5978 scope.go:122] "RemoveContainer" containerID="5291026b01c8c21bc6c65025079a3988ac597fc41b075adb223b558ebe42acf6" Oct 31 01:35:49.483978 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:49.483959 5978 scope.go:122] "RemoveContainer" containerID="456639ca91f7c50c39669dae8ed54065e74b9dad4c1851da966e38ab32f10346" Oct 31 01:35:49.734919 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:49.734834 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:51.255060 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:51.255019 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:52.399235 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:52.399198 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:52.893541 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:52.893500 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:54.502297 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:54.502255 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:55.973643 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:55.973610 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:56.809578 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:56.809537 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:57.593935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:57.593892 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:59.468888 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:35:59.468848 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:00.991863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:00.991824 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:01.729167 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:01.729129 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph"] Oct 31 01:36:01.862894 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:01.862855 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:02.705499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:02.705464 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:03.181358 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:36:03.181320 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:36:04.300684 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:04.300647 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:05.875789 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:05.875752 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:07.181276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.181239 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8"] Oct 31 01:36:07.181838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.181557 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" podUID="bac09980-4503-41db-acf7-3e24a2a44c46" containerName="maas-api" containerID="cri-o://6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9" gracePeriod=30 Oct 31 01:36:07.288548 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.288502 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:07.458137 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.458051 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:07.603395 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.603359 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:07.647867 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.647835 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:36:07.733880 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.733795 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-maas-api-tls\") pod \"bac09980-4503-41db-acf7-3e24a2a44c46\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " Oct 31 01:36:07.733880 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.733862 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac09980-4503-41db-acf7-3e24a2a44c46-kube-api-access-p6k8j\" (UniqueName: \"kubernetes.io/projected/bac09980-4503-41db-acf7-3e24a2a44c46-kube-api-access-p6k8j\") pod \"bac09980-4503-41db-acf7-3e24a2a44c46\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " Oct 31 01:36:07.734091 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.733987 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-metrics-certs\") pod \"bac09980-4503-41db-acf7-3e24a2a44c46\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " Oct 31 01:36:07.734091 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.734020 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac09980-4503-41db-acf7-3e24a2a44c46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac09980-4503-41db-acf7-3e24a2a44c46-openshift-service-ca\") pod \"bac09980-4503-41db-acf7-3e24a2a44c46\" (UID: \"bac09980-4503-41db-acf7-3e24a2a44c46\") " Oct 31 01:36:07.734414 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.734382 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac09980-4503-41db-acf7-3e24a2a44c46-openshift-service-ca" pod "bac09980-4503-41db-acf7-3e24a2a44c46" (UID: "bac09980-4503-41db-acf7-3e24a2a44c46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:36:07.736042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.735998 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-maas-api-tls" pod "bac09980-4503-41db-acf7-3e24a2a44c46" (UID: "bac09980-4503-41db-acf7-3e24a2a44c46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:36:07.736542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.736039 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac09980-4503-41db-acf7-3e24a2a44c46-kube-api-access-p6k8j" pod "bac09980-4503-41db-acf7-3e24a2a44c46" (UID: "bac09980-4503-41db-acf7-3e24a2a44c46"). InnerVolumeSpecName "kube-api-access-p6k8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:36:07.736542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.736091 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-metrics-certs" pod "bac09980-4503-41db-acf7-3e24a2a44c46" (UID: "bac09980-4503-41db-acf7-3e24a2a44c46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:36:07.835369 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.835329 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:07.835369 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.835369 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6k8j\" (UniqueName: \"kubernetes.io/projected/bac09980-4503-41db-acf7-3e24a2a44c46-kube-api-access-p6k8j\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:07.835602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.835386 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac09980-4503-41db-acf7-3e24a2a44c46-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:07.835602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:07.835401 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac09980-4503-41db-acf7-3e24a2a44c46-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:08.295485 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.295434 5978 generic.go:356] "Generic (PLEG): container finished" podID="bac09980-4503-41db-acf7-3e24a2a44c46" containerID="6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9" exitCode=0 Oct 31 01:36:08.295485 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.295482 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" event={"ID":"bac09980-4503-41db-acf7-3e24a2a44c46","Type":"ContainerDied","Data":"6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9"} Oct 31 01:36:08.296003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.295505 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" Oct 31 01:36:08.296003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.295622 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8" event={"ID":"bac09980-4503-41db-acf7-3e24a2a44c46","Type":"ContainerDied","Data":"73627b87345f67df0c37267af1f5957319f8d24827c9a68213361890dac50fb5"} Oct 31 01:36:08.296003 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.295650 5978 scope.go:122] "RemoveContainer" containerID="6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9" Oct 31 01:36:08.306109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.306089 5978 scope.go:122] "RemoveContainer" containerID="6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9" Oct 31 01:36:08.306362 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:36:08.306344 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9\": container with ID starting with 6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9 not found: ID does not exist" containerID="6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9" Oct 31 01:36:08.306419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.306374 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9"} err="failed to get container status \"6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9\": rpc error: code = NotFound desc = could not find container \"6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9\": container with ID starting with 6e4a2afa154e09b55e563b65ebc618cea9f463a09a98abac41caee3f473af8a9 not found: ID does not exist" Oct 31 01:36:08.339102 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.339073 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8"] Oct 31 01:36:08.348854 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.348827 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e17bf-86c64457ff-t2kv8"] Oct 31 01:36:08.424031 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.423997 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac09980-4503-41db-acf7-3e24a2a44c46" path="/var/lib/kubelet/pods/bac09980-4503-41db-acf7-3e24a2a44c46/volumes" Oct 31 01:36:08.724582 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.724552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:08.930964 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:08.930908 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:09.556780 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:09.556740 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:15.291188 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:15.291148 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:15.444076 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:15.444041 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:15.654499 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:15.654448 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:16.750889 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:16.750839 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" podUID="00a38c67-6349-4082-bd86-80e1a2d6724a" containerName="main" containerID="cri-o://047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495" gracePeriod=45 Oct 31 01:36:17.004199 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.004127 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:36:17.131627 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.131596 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-home\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-home\") pod \"00a38c67-6349-4082-bd86-80e1a2d6724a\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " Oct 31 01:36:17.131840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.131641 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a38c67-6349-4082-bd86-80e1a2d6724a-kube-api-access-7m6zk\" (UniqueName: \"kubernetes.io/projected/00a38c67-6349-4082-bd86-80e1a2d6724a-kube-api-access-7m6zk\") pod \"00a38c67-6349-4082-bd86-80e1a2d6724a\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " Oct 31 01:36:17.131840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.131690 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-dshm\") pod \"00a38c67-6349-4082-bd86-80e1a2d6724a\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " Oct 31 01:36:17.131840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.131730 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-model-cache\") pod \"00a38c67-6349-4082-bd86-80e1a2d6724a\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " Oct 31 01:36:17.131840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.131832 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a38c67-6349-4082-bd86-80e1a2d6724a-tls-certs\" (UniqueName: \"kubernetes.io/secret/00a38c67-6349-4082-bd86-80e1a2d6724a-tls-certs\") pod \"00a38c67-6349-4082-bd86-80e1a2d6724a\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " Oct 31 01:36:17.132086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.131913 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-tmp-dir\") pod \"00a38c67-6349-4082-bd86-80e1a2d6724a\" (UID: \"00a38c67-6349-4082-bd86-80e1a2d6724a\") " Oct 31 01:36:17.132086 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.131922 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-home" pod "00a38c67-6349-4082-bd86-80e1a2d6724a" (UID: "00a38c67-6349-4082-bd86-80e1a2d6724a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:36:17.132205 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.132082 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-model-cache" pod "00a38c67-6349-4082-bd86-80e1a2d6724a" (UID: "00a38c67-6349-4082-bd86-80e1a2d6724a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:36:17.132340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.132255 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-tmp-dir" pod "00a38c67-6349-4082-bd86-80e1a2d6724a" (UID: "00a38c67-6349-4082-bd86-80e1a2d6724a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:36:17.132616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.132425 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:17.132616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.132448 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:17.132616 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.132527 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:17.134254 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.134234 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a38c67-6349-4082-bd86-80e1a2d6724a-kube-api-access-7m6zk" pod "00a38c67-6349-4082-bd86-80e1a2d6724a" (UID: "00a38c67-6349-4082-bd86-80e1a2d6724a"). InnerVolumeSpecName "kube-api-access-7m6zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:36:17.134542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.134515 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-dshm" pod "00a38c67-6349-4082-bd86-80e1a2d6724a" (UID: "00a38c67-6349-4082-bd86-80e1a2d6724a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:36:17.134542 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.134527 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a38c67-6349-4082-bd86-80e1a2d6724a-tls-certs" pod "00a38c67-6349-4082-bd86-80e1a2d6724a" (UID: "00a38c67-6349-4082-bd86-80e1a2d6724a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:36:17.233343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.233308 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a38c67-6349-4082-bd86-80e1a2d6724a-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:17.233343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.233339 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m6zk\" (UniqueName: \"kubernetes.io/projected/00a38c67-6349-4082-bd86-80e1a2d6724a-kube-api-access-7m6zk\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:17.233343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.233350 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a38c67-6349-4082-bd86-80e1a2d6724a-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:36:17.349727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.349695 5978 generic.go:356] "Generic (PLEG): container finished" podID="00a38c67-6349-4082-bd86-80e1a2d6724a" containerID="047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495" exitCode=0 Oct 31 01:36:17.349910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.349747 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" Oct 31 01:36:17.349910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.349770 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" event={"ID":"00a38c67-6349-4082-bd86-80e1a2d6724a","Type":"ContainerDied","Data":"047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495"} Oct 31 01:36:17.349910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.349803 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph" event={"ID":"00a38c67-6349-4082-bd86-80e1a2d6724a","Type":"ContainerDied","Data":"75c27a9e98390a1748602614420c2d7993cb811f2734977cf9e7659c58fa42af"} Oct 31 01:36:17.349910 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.349821 5978 scope.go:122] "RemoveContainer" containerID="047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495" Oct 31 01:36:17.359198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.359168 5978 scope.go:122] "RemoveContainer" containerID="047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495" Oct 31 01:36:17.359615 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:36:17.359593 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495\": container with ID starting with 047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495 not found: ID does not exist" containerID="047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495" Oct 31 01:36:17.359666 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.359626 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495"} err="failed to get container status \"047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495\": rpc error: code = NotFound desc = could not find container \"047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495\": container with ID starting with 047f49cdf2b982cf379f207b7af61da99f650ecedb57714fb2dbda6050096495 not found: ID does not exist" Oct 31 01:36:17.394764 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.394726 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph"] Oct 31 01:36:17.396464 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:17.396432 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2e17bf/simulator-single-main-2e17bf-kserve-6b6f6c88dc-j2kph"] Oct 31 01:36:18.424327 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:18.424296 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a38c67-6349-4082-bd86-80e1a2d6724a" path="/var/lib/kubelet/pods/00a38c67-6349-4082-bd86-80e1a2d6724a/volumes" Oct 31 01:36:21.361655 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.361609 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c"] Oct 31 01:36:21.362370 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.362339 5978 cpu_manager.go:414] "containerMap: removing container" podUID="bac09980-4503-41db-acf7-3e24a2a44c46" containerName="maas-api" Oct 31 01:36:21.362370 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.362365 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac09980-4503-41db-acf7-3e24a2a44c46" containerName="maas-api" Oct 31 01:36:21.362611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.362387 5978 cpu_manager.go:414] "containerMap: removing container" podUID="00a38c67-6349-4082-bd86-80e1a2d6724a" containerName="main" Oct 31 01:36:21.362611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.362395 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a38c67-6349-4082-bd86-80e1a2d6724a" containerName="main" Oct 31 01:36:21.362611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.362589 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac09980-4503-41db-acf7-3e24a2a44c46" containerName="maas-api" Oct 31 01:36:21.362611 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.362607 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a38c67-6349-4082-bd86-80e1a2d6724a" containerName="main" Oct 31 01:36:21.367578 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.367550 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.392239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.392208 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-51f40966-kserve-self-signed-certs\"" Oct 31 01:36:21.394350 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.394316 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c"] Oct 31 01:36:21.465956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.465919 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:21.479211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.479171 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-model-cache\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.479211 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.479214 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tmp-dir\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.479415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.479302 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-dshm\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.479415 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.479374 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-home\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.479530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.479426 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqbng\" (UniqueName: \"kubernetes.io/projected/82d7378a-6e40-483a-bd84-e6dbfa2e234e-kube-api-access-wqbng\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.479568 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.479548 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tls-certs\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.580306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.580243 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-model-cache\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.580573 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.580341 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tmp-dir\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.580573 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.580394 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-dshm\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.580573 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.580422 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-home\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.580788 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.580573 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqbng\" (UniqueName: \"kubernetes.io/projected/82d7378a-6e40-483a-bd84-e6dbfa2e234e-kube-api-access-wqbng\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.580788 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.580696 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tls-certs\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.581592 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.581518 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-home\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.581887 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.581648 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tmp-dir\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.581887 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.581843 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-model-cache\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.583581 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.583550 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-dshm\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.583924 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.583906 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tls-certs\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.615735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.615656 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqbng\" (UniqueName: \"kubernetes.io/projected/82d7378a-6e40-483a-bd84-e6dbfa2e234e-kube-api-access-wqbng\") pod \"e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.684426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.684383 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:21.982760 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:21.982706 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c"] Oct 31 01:36:21.985366 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:36:21.985340 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82d7378a_6e40_483a_bd84_e6dbfa2e234e.slice/crio-f74ca4846741e1eeaa626c6a25ca1616d588f873ce584a8983f32ddedcc66b38 WatchSource:0}: Error finding container f74ca4846741e1eeaa626c6a25ca1616d588f873ce584a8983f32ddedcc66b38: Status 404 returned error can't find the container with id f74ca4846741e1eeaa626c6a25ca1616d588f873ce584a8983f32ddedcc66b38 Oct 31 01:36:22.380070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:22.380036 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" event={"ID":"82d7378a-6e40-483a-bd84-e6dbfa2e234e","Type":"ContainerStarted","Data":"30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389"} Oct 31 01:36:22.380070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:22.380079 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" event={"ID":"82d7378a-6e40-483a-bd84-e6dbfa2e234e","Type":"ContainerStarted","Data":"f74ca4846741e1eeaa626c6a25ca1616d588f873ce584a8983f32ddedcc66b38"} Oct 31 01:36:22.421750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:22.421692 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" podStartSLOduration=1.421672794 podStartE2EDuration="1.421672794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:22.418525579 +0000 UTC m=+4600.116639345" watchObservedRunningTime="2026-10-31 01:36:22.421672794 +0000 UTC m=+4600.119786562" Oct 31 01:36:22.470115 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:22.470076 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:22.685369 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:22.685275 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:31.685217 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:31.685118 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:31.697941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:31.697907 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:32.494661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:32.494629 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:36:33.182229 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:36:33.182189 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:36:37.765285 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:37.765247 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:38.690964 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:38.690916 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:39.322938 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:39.322901 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:49.515033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:49.514991 5978 scope.go:122] "RemoveContainer" containerID="0604cae334d75c796e32dbe1e5aa7c5ea77ad14b4f304374e8cd0b683189da58" Oct 31 01:36:49.524833 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:49.524805 5978 scope.go:122] "RemoveContainer" containerID="f070d00bedf9428f4e95b2986e2ecb5f71822f898f5149c6e46155da09ce65aa" Oct 31 01:36:52.492597 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:52.492560 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:53.986307 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:36:53.986269 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:01.245389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.242303 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx"] Oct 31 01:37:01.247630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.247598 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.260388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.260356 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:37:01.260388 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.260380 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:37:01.276501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.270692 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:37:01.276501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.270850 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:37:01.276501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.273830 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx"] Oct 31 01:37:01.405863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.405816 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-kube-api-access-5n6x4\") pod \"maas-api-revoke-sub-452b6d7b-5gljx\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.406106 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.405983 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5gljx\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.406106 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.406070 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5gljx\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.507197 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.507088 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5gljx\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.507400 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.507196 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5gljx\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.507512 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.507397 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-kube-api-access-5n6x4\") pod \"maas-api-revoke-sub-452b6d7b-5gljx\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.508426 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.508389 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5gljx\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.510351 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.510321 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5gljx\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.529819 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.529780 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-kube-api-access-5n6x4\") pod \"maas-api-revoke-sub-452b6d7b-5gljx\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.568633 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.568588 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:01.935660 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:01.935607 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx"] Oct 31 01:37:02.801779 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:02.801667 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" event={"ID":"b0f9c96b-dbb2-455d-bbb3-83de84ea7932","Type":"ContainerStarted","Data":"9d712ddf5edb62a2e5a9321874ca60e301067d474e23e19ac73c68511254304f"} Oct 31 01:37:02.802400 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:02.802372 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" event={"ID":"b0f9c96b-dbb2-455d-bbb3-83de84ea7932","Type":"ContainerStarted","Data":"039479583cd56f6a7552de18070fcff0972d33ef85c034533a9ce8c621114620"} Oct 31 01:37:03.176654 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:03.176614 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:03.354477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:03.336176 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:03.363740 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:37:03.363694 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:37:03.839657 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:03.839619 5978 generic.go:356] "Generic (PLEG): container finished" podID="b0f9c96b-dbb2-455d-bbb3-83de84ea7932" containerID="9d712ddf5edb62a2e5a9321874ca60e301067d474e23e19ac73c68511254304f" exitCode=0 Oct 31 01:37:03.840396 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:03.840366 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" event={"ID":"b0f9c96b-dbb2-455d-bbb3-83de84ea7932","Type":"ContainerDied","Data":"9d712ddf5edb62a2e5a9321874ca60e301067d474e23e19ac73c68511254304f"} Oct 31 01:37:04.991650 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:04.991622 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:05.021465 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.021429 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-kube-api-access-5n6x4\") pod \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " Oct 31 01:37:05.021660 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.021583 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-openshift-ca\") pod \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " Oct 31 01:37:05.021660 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.021638 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-xks-ca\") pod \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\" (UID: \"b0f9c96b-dbb2-455d-bbb3-83de84ea7932\") " Oct 31 01:37:05.021925 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.021900 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-openshift-ca" pod "b0f9c96b-dbb2-455d-bbb3-83de84ea7932" (UID: "b0f9c96b-dbb2-455d-bbb3-83de84ea7932"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:37:05.023663 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.023641 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-kube-api-access-5n6x4" pod "b0f9c96b-dbb2-455d-bbb3-83de84ea7932" (UID: "b0f9c96b-dbb2-455d-bbb3-83de84ea7932"). InnerVolumeSpecName "kube-api-access-5n6x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:05.023731 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.023655 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-xks-ca" pod "b0f9c96b-dbb2-455d-bbb3-83de84ea7932" (UID: "b0f9c96b-dbb2-455d-bbb3-83de84ea7932"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:05.123276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.123187 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:05.123276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.123223 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:05.123276 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.123232 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/b0f9c96b-dbb2-455d-bbb3-83de84ea7932-kube-api-access-5n6x4\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:05.852552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.852518 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" Oct 31 01:37:05.852552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.852543 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx" event={"ID":"b0f9c96b-dbb2-455d-bbb3-83de84ea7932","Type":"ContainerDied","Data":"039479583cd56f6a7552de18070fcff0972d33ef85c034533a9ce8c621114620"} Oct 31 01:37:05.852794 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:05.852604 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="039479583cd56f6a7552de18070fcff0972d33ef85c034533a9ce8c621114620" Oct 31 01:37:12.437432 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:12.437395 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c"] Oct 31 01:37:12.499202 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:12.499148 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:16.112478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:16.112395 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:19.704795 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.704760 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b"] Oct 31 01:37:19.705362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.705340 5978 cpu_manager.go:414] "containerMap: removing container" podUID="b0f9c96b-dbb2-455d-bbb3-83de84ea7932" containerName="revoke-keys" Oct 31 01:37:19.705429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.705364 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f9c96b-dbb2-455d-bbb3-83de84ea7932" containerName="revoke-keys" Oct 31 01:37:19.705500 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.705445 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f9c96b-dbb2-455d-bbb3-83de84ea7932" containerName="revoke-keys" Oct 31 01:37:19.708886 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.708864 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:19.724809 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.724778 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:37:19.727100 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.727077 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:37:19.734966 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.734930 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:37:19.734966 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.734955 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:37:19.737074 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.737044 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b"] Oct 31 01:37:19.819534 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.819495 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vqq8b\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:19.819766 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.819551 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vqq8b\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:19.819766 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.819676 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntshq\" (UniqueName: \"kubernetes.io/projected/c900896d-c6eb-43cd-befd-93a5c5c758e9-kube-api-access-ntshq\") pod \"maas-api-revoke-sub-3e31a506-vqq8b\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:19.921340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.921304 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vqq8b\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:19.921571 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.921360 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vqq8b\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:19.921571 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.921419 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntshq\" (UniqueName: \"kubernetes.io/projected/c900896d-c6eb-43cd-befd-93a5c5c758e9-kube-api-access-ntshq\") pod \"maas-api-revoke-sub-3e31a506-vqq8b\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:19.922115 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.922093 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vqq8b\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:19.923738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.923710 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vqq8b\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:19.942820 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:19.942786 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntshq\" (UniqueName: \"kubernetes.io/projected/c900896d-c6eb-43cd-befd-93a5c5c758e9-kube-api-access-ntshq\") pod \"maas-api-revoke-sub-3e31a506-vqq8b\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:20.024316 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:20.024226 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:20.334843 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:20.334480 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b"] Oct 31 01:37:20.338876 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:37:20.338835 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc900896d_c6eb_43cd_befd_93a5c5c758e9.slice/crio-db44e9c5f139d7145cefaa254c49155303ee0733ad7c59e612dab3f7b109028e WatchSource:0}: Error finding container db44e9c5f139d7145cefaa254c49155303ee0733ad7c59e612dab3f7b109028e: Status 404 returned error can't find the container with id db44e9c5f139d7145cefaa254c49155303ee0733ad7c59e612dab3f7b109028e Oct 31 01:37:20.991343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:20.991251 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" event={"ID":"c900896d-c6eb-43cd-befd-93a5c5c758e9","Type":"ContainerStarted","Data":"9d191ccbbfcac7336f5803a2816a66b1d2c0ebbdeab2f4e4594afee0ea7c70a8"} Oct 31 01:37:20.991343 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:20.991318 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" event={"ID":"c900896d-c6eb-43cd-befd-93a5c5c758e9","Type":"ContainerStarted","Data":"db44e9c5f139d7145cefaa254c49155303ee0733ad7c59e612dab3f7b109028e"} Oct 31 01:37:22.005759 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:22.005693 5978 generic.go:356] "Generic (PLEG): container finished" podID="c900896d-c6eb-43cd-befd-93a5c5c758e9" containerID="9d191ccbbfcac7336f5803a2816a66b1d2c0ebbdeab2f4e4594afee0ea7c70a8" exitCode=0 Oct 31 01:37:22.006286 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:22.005805 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" event={"ID":"c900896d-c6eb-43cd-befd-93a5c5c758e9","Type":"ContainerDied","Data":"9d191ccbbfcac7336f5803a2816a66b1d2c0ebbdeab2f4e4594afee0ea7c70a8"} Oct 31 01:37:23.144913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.144885 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:23.258358 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.258316 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-xks-ca\") pod \"c900896d-c6eb-43cd-befd-93a5c5c758e9\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " Oct 31 01:37:23.258586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.258394 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c900896d-c6eb-43cd-befd-93a5c5c758e9-kube-api-access-ntshq\" (UniqueName: \"kubernetes.io/projected/c900896d-c6eb-43cd-befd-93a5c5c758e9-kube-api-access-ntshq\") pod \"c900896d-c6eb-43cd-befd-93a5c5c758e9\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " Oct 31 01:37:23.258586 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.258488 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-openshift-ca\") pod \"c900896d-c6eb-43cd-befd-93a5c5c758e9\" (UID: \"c900896d-c6eb-43cd-befd-93a5c5c758e9\") " Oct 31 01:37:23.258814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.258789 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-openshift-ca" pod "c900896d-c6eb-43cd-befd-93a5c5c758e9" (UID: "c900896d-c6eb-43cd-befd-93a5c5c758e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:37:23.258973 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.258958 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:23.260633 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.260603 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c900896d-c6eb-43cd-befd-93a5c5c758e9-kube-api-access-ntshq" pod "c900896d-c6eb-43cd-befd-93a5c5c758e9" (UID: "c900896d-c6eb-43cd-befd-93a5c5c758e9"). InnerVolumeSpecName "kube-api-access-ntshq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:23.260748 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.260646 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-xks-ca" pod "c900896d-c6eb-43cd-befd-93a5c5c758e9" (UID: "c900896d-c6eb-43cd-befd-93a5c5c758e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:23.359913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.359870 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c900896d-c6eb-43cd-befd-93a5c5c758e9-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:23.359913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:23.359906 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntshq\" (UniqueName: \"kubernetes.io/projected/c900896d-c6eb-43cd-befd-93a5c5c758e9-kube-api-access-ntshq\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:24.017783 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:24.017743 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" event={"ID":"c900896d-c6eb-43cd-befd-93a5c5c758e9","Type":"ContainerDied","Data":"db44e9c5f139d7145cefaa254c49155303ee0733ad7c59e612dab3f7b109028e"} Oct 31 01:37:24.017783 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:24.017785 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b" Oct 31 01:37:24.018015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:24.017823 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db44e9c5f139d7145cefaa254c49155303ee0733ad7c59e612dab3f7b109028e" Oct 31 01:37:27.458337 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.458276 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" podUID="82d7378a-6e40-483a-bd84-e6dbfa2e234e" containerName="main" containerID="cri-o://30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389" gracePeriod=45 Oct 31 01:37:27.484603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.484570 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" podUID="82d7378a-6e40-483a-bd84-e6dbfa2e234e" containerName="main" probeResult="failure" output="Get \"https://10.134.1.1:8000/ready\": dial tcp 10.134.1.1:8000: connect: connection refused" Oct 31 01:37:27.715846 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.715779 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:37:27.806899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.806868 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-dshm\") pod \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " Oct 31 01:37:27.806899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.806906 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tmp-dir\") pod \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " Oct 31 01:37:27.807149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.806950 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-home\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-home\") pod \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " Oct 31 01:37:27.807149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.807075 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tls-certs\" (UniqueName: \"kubernetes.io/secret/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tls-certs\") pod \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " Oct 31 01:37:27.807149 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.807100 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d7378a-6e40-483a-bd84-e6dbfa2e234e-kube-api-access-wqbng\" (UniqueName: \"kubernetes.io/projected/82d7378a-6e40-483a-bd84-e6dbfa2e234e-kube-api-access-wqbng\") pod \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " Oct 31 01:37:27.807327 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.807152 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-model-cache\") pod \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\" (UID: \"82d7378a-6e40-483a-bd84-e6dbfa2e234e\") " Oct 31 01:37:27.807327 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.807216 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tmp-dir" pod "82d7378a-6e40-483a-bd84-e6dbfa2e234e" (UID: "82d7378a-6e40-483a-bd84-e6dbfa2e234e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:27.807327 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.807300 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-home" pod "82d7378a-6e40-483a-bd84-e6dbfa2e234e" (UID: "82d7378a-6e40-483a-bd84-e6dbfa2e234e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:27.807527 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.807493 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-model-cache" pod "82d7378a-6e40-483a-bd84-e6dbfa2e234e" (UID: "82d7378a-6e40-483a-bd84-e6dbfa2e234e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:27.807643 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.807625 5978 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-model-cache\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:27.807689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.807649 5978 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tmp-dir\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:27.807689 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.807661 5978 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-home\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:27.809125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.809098 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-dshm" pod "82d7378a-6e40-483a-bd84-e6dbfa2e234e" (UID: "82d7378a-6e40-483a-bd84-e6dbfa2e234e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:27.809239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.809183 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tls-certs" pod "82d7378a-6e40-483a-bd84-e6dbfa2e234e" (UID: "82d7378a-6e40-483a-bd84-e6dbfa2e234e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:27.809239 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.809199 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d7378a-6e40-483a-bd84-e6dbfa2e234e-kube-api-access-wqbng" pod "82d7378a-6e40-483a-bd84-e6dbfa2e234e" (UID: "82d7378a-6e40-483a-bd84-e6dbfa2e234e"). InnerVolumeSpecName "kube-api-access-wqbng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:27.908708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.908673 5978 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82d7378a-6e40-483a-bd84-e6dbfa2e234e-tls-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:27.908708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.908703 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqbng\" (UniqueName: \"kubernetes.io/projected/82d7378a-6e40-483a-bd84-e6dbfa2e234e-kube-api-access-wqbng\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:27.908708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:27.908713 5978 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d7378a-6e40-483a-bd84-e6dbfa2e234e-dshm\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:28.040580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.040494 5978 generic.go:356] "Generic (PLEG): container finished" podID="82d7378a-6e40-483a-bd84-e6dbfa2e234e" containerID="30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389" exitCode=0 Oct 31 01:37:28.040580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.040542 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" Oct 31 01:37:28.040778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.040582 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" event={"ID":"82d7378a-6e40-483a-bd84-e6dbfa2e234e","Type":"ContainerDied","Data":"30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389"} Oct 31 01:37:28.040778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.040630 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c" event={"ID":"82d7378a-6e40-483a-bd84-e6dbfa2e234e","Type":"ContainerDied","Data":"f74ca4846741e1eeaa626c6a25ca1616d588f873ce584a8983f32ddedcc66b38"} Oct 31 01:37:28.040778 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.040653 5978 scope.go:122] "RemoveContainer" containerID="30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389" Oct 31 01:37:28.050482 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.050447 5978 scope.go:122] "RemoveContainer" containerID="30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389" Oct 31 01:37:28.050776 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:37:28.050756 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389\": container with ID starting with 30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389 not found: ID does not exist" containerID="30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389" Oct 31 01:37:28.050838 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.050787 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389"} err="failed to get container status \"30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389\": rpc error: code = NotFound desc = could not find container \"30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389\": container with ID starting with 30cf72be90c2df228fda49a53d1574a3b92164d717a169b7f26674b52bd96389 not found: ID does not exist" Oct 31 01:37:28.085883 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.085851 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c"] Oct 31 01:37:28.090125 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.090102 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-51f40966-kserve-54998fc8d5-d5h9c"] Oct 31 01:37:28.423992 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:28.423958 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d7378a-6e40-483a-bd84-e6dbfa2e234e" path="/var/lib/kubelet/pods/82d7378a-6e40-483a-bd84-e6dbfa2e234e/volumes" Oct 31 01:37:33.374476 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:37:33.373760 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:37:35.029077 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:35.029029 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2"] Oct 31 01:37:35.038314 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:35.038278 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7n7c2"] Oct 31 01:37:36.442245 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:36.442202 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b962734d-5d45-49fa-8c0c-2037e11c08da" path="/var/lib/kubelet/pods/b962734d-5d45-49fa-8c0c-2037e11c08da/volumes" Oct 31 01:37:43.285089 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.285052 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb"] Oct 31 01:37:43.285751 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.285732 5978 cpu_manager.go:414] "containerMap: removing container" podUID="82d7378a-6e40-483a-bd84-e6dbfa2e234e" containerName="main" Oct 31 01:37:43.285836 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.285753 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d7378a-6e40-483a-bd84-e6dbfa2e234e" containerName="main" Oct 31 01:37:43.285836 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.285766 5978 cpu_manager.go:414] "containerMap: removing container" podUID="c900896d-c6eb-43cd-befd-93a5c5c758e9" containerName="revoke-keys" Oct 31 01:37:43.285836 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.285772 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c900896d-c6eb-43cd-befd-93a5c5c758e9" containerName="revoke-keys" Oct 31 01:37:43.285945 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.285864 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d7378a-6e40-483a-bd84-e6dbfa2e234e" containerName="main" Oct 31 01:37:43.285945 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.285873 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="c900896d-c6eb-43cd-befd-93a5c5c758e9" containerName="revoke-keys" Oct 31 01:37:43.290604 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.290583 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.303114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.303088 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:37:43.303114 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.303102 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:37:43.312837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.312808 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:37:43.313085 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.313068 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4n7v\"" Oct 31 01:37:43.315146 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.315118 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb"] Oct 31 01:37:43.397424 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.397383 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cptw\" (UniqueName: \"kubernetes.io/projected/a52333b1-a32e-487d-85a6-76a4e130dc5a-kube-api-access-8cptw\") pod \"maas-api-revoke-sub-03496072-hl7hb\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.397637 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.397551 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03496072-hl7hb\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.397637 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.397579 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03496072-hl7hb\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.498725 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.498687 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03496072-hl7hb\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.498933 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.498734 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03496072-hl7hb\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.498933 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.498786 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cptw\" (UniqueName: \"kubernetes.io/projected/a52333b1-a32e-487d-85a6-76a4e130dc5a-kube-api-access-8cptw\") pod \"maas-api-revoke-sub-03496072-hl7hb\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.499541 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.499515 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03496072-hl7hb\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.501108 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.501087 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03496072-hl7hb\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.520674 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.520639 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cptw\" (UniqueName: \"kubernetes.io/projected/a52333b1-a32e-487d-85a6-76a4e130dc5a-kube-api-access-8cptw\") pod \"maas-api-revoke-sub-03496072-hl7hb\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.603577 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.603538 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:43.901338 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:43.901312 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb"] Oct 31 01:37:43.903819 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:37:43.903777 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52333b1_a32e_487d_85a6_76a4e130dc5a.slice/crio-56f642de2d2a04ad30c130b95ab5ba9f4c0ddc87cb5d68436dc461425f1c4783 WatchSource:0}: Error finding container 56f642de2d2a04ad30c130b95ab5ba9f4c0ddc87cb5d68436dc461425f1c4783: Status 404 returned error can't find the container with id 56f642de2d2a04ad30c130b95ab5ba9f4c0ddc87cb5d68436dc461425f1c4783 Oct 31 01:37:44.237138 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:44.237109 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" event={"ID":"a52333b1-a32e-487d-85a6-76a4e130dc5a","Type":"ContainerStarted","Data":"697d5b45e718603dbbd4ac555c1554502b45ed7ebd1ce01e4626f59dc5b91270"} Oct 31 01:37:44.237291 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:44.237162 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" event={"ID":"a52333b1-a32e-487d-85a6-76a4e130dc5a","Type":"ContainerStarted","Data":"56f642de2d2a04ad30c130b95ab5ba9f4c0ddc87cb5d68436dc461425f1c4783"} Oct 31 01:37:44.273352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:44.273283 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" podStartSLOduration=1.27326903 podStartE2EDuration="1.27326903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:44.273205885 +0000 UTC m=+4681.971319660" watchObservedRunningTime="2026-10-31 01:37:44.27326903 +0000 UTC m=+4681.971382794" Oct 31 01:37:45.243770 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:45.243736 5978 generic.go:356] "Generic (PLEG): container finished" podID="a52333b1-a32e-487d-85a6-76a4e130dc5a" containerID="697d5b45e718603dbbd4ac555c1554502b45ed7ebd1ce01e4626f59dc5b91270" exitCode=0 Oct 31 01:37:45.244194 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:45.243826 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" event={"ID":"a52333b1-a32e-487d-85a6-76a4e130dc5a","Type":"ContainerDied","Data":"697d5b45e718603dbbd4ac555c1554502b45ed7ebd1ce01e4626f59dc5b91270"} Oct 31 01:37:46.383604 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.383576 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:46.529231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.529137 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-xks-ca\") pod \"a52333b1-a32e-487d-85a6-76a4e130dc5a\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " Oct 31 01:37:46.529231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.529231 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52333b1-a32e-487d-85a6-76a4e130dc5a-kube-api-access-8cptw\" (UniqueName: \"kubernetes.io/projected/a52333b1-a32e-487d-85a6-76a4e130dc5a-kube-api-access-8cptw\") pod \"a52333b1-a32e-487d-85a6-76a4e130dc5a\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " Oct 31 01:37:46.529543 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.529264 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-openshift-ca\") pod \"a52333b1-a32e-487d-85a6-76a4e130dc5a\" (UID: \"a52333b1-a32e-487d-85a6-76a4e130dc5a\") " Oct 31 01:37:46.529722 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.529693 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-openshift-ca" pod "a52333b1-a32e-487d-85a6-76a4e130dc5a" (UID: "a52333b1-a32e-487d-85a6-76a4e130dc5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:37:46.531304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.531279 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-xks-ca" pod "a52333b1-a32e-487d-85a6-76a4e130dc5a" (UID: "a52333b1-a32e-487d-85a6-76a4e130dc5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:46.531448 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.531340 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52333b1-a32e-487d-85a6-76a4e130dc5a-kube-api-access-8cptw" pod "a52333b1-a32e-487d-85a6-76a4e130dc5a" (UID: "a52333b1-a32e-487d-85a6-76a4e130dc5a"). InnerVolumeSpecName "kube-api-access-8cptw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:46.630735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.630691 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-xks-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:46.630735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.630729 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cptw\" (UniqueName: \"kubernetes.io/projected/a52333b1-a32e-487d-85a6-76a4e130dc5a-kube-api-access-8cptw\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:46.630735 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:46.630740 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52333b1-a32e-487d-85a6-76a4e130dc5a-maas-api-openshift-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:37:47.255692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:47.255654 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" event={"ID":"a52333b1-a32e-487d-85a6-76a4e130dc5a","Type":"ContainerDied","Data":"56f642de2d2a04ad30c130b95ab5ba9f4c0ddc87cb5d68436dc461425f1c4783"} Oct 31 01:37:47.255692 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:47.255679 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb" Oct 31 01:37:47.255917 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:47.255704 5978 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56f642de2d2a04ad30c130b95ab5ba9f4c0ddc87cb5d68436dc461425f1c4783" Oct 31 01:37:52.610113 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:52.610073 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:55.797127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.797042 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq"] Oct 31 01:37:55.797754 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.797733 5978 cpu_manager.go:414] "containerMap: removing container" podUID="a52333b1-a32e-487d-85a6-76a4e130dc5a" containerName="revoke-keys" Oct 31 01:37:55.797839 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.797758 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52333b1-a32e-487d-85a6-76a4e130dc5a" containerName="revoke-keys" Oct 31 01:37:55.797900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.797855 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52333b1-a32e-487d-85a6-76a4e130dc5a" containerName="revoke-keys" Oct 31 01:37:55.801136 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.801111 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.823768 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.823733 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:37:55.823981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.823804 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cckh7\"" Oct 31 01:37:55.823981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.823853 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:37:55.823981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.823926 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:37:55.823981 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.823955 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 01:37:55.826389 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.826336 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq"] Oct 31 01:37:55.832773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.832738 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-openshift-service-ca\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.832950 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.832900 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q57gp\" (UniqueName: \"kubernetes.io/projected/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-kube-api-access-q57gp\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.833015 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.832973 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-metrics-certs\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.833077 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.833054 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-maas-api-tls\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.934475 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.934427 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q57gp\" (UniqueName: \"kubernetes.io/projected/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-kube-api-access-q57gp\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.934708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.934538 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-metrics-certs\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.934708 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.934572 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-maas-api-tls\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.934832 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.934709 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-openshift-service-ca\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.935325 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.935287 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-openshift-service-ca\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.937026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.937001 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-maas-api-tls\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.937151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.937045 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-metrics-certs\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:55.955375 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:55.955340 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q57gp\" (UniqueName: \"kubernetes.io/projected/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-kube-api-access-q57gp\") pod \"maas-api-6fb4ff9887-4wgbq\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:56.115336 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:56.115297 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:56.409041 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:56.408958 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq"] Oct 31 01:37:56.411119 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:37:56.411090 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a449cbc_1662_4a50_adbf_3d5d0a18d86a.slice/crio-0c664c4fd65d3c62c0956de0a55b7020057bc7e8ad365cd779ac43655ca255ca WatchSource:0}: Error finding container 0c664c4fd65d3c62c0956de0a55b7020057bc7e8ad365cd779ac43655ca255ca: Status 404 returned error can't find the container with id 0c664c4fd65d3c62c0956de0a55b7020057bc7e8ad365cd779ac43655ca255ca Oct 31 01:37:56.412559 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:56.412542 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:37:57.309392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:57.309292 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" event={"ID":"0a449cbc-1662-4a50-adbf-3d5d0a18d86a","Type":"ContainerStarted","Data":"db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa"} Oct 31 01:37:57.309815 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:57.309397 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" event={"ID":"0a449cbc-1662-4a50-adbf-3d5d0a18d86a","Type":"ContainerStarted","Data":"0c664c4fd65d3c62c0956de0a55b7020057bc7e8ad365cd779ac43655ca255ca"} Oct 31 01:37:57.309815 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:57.309557 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:37:57.347531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:37:57.347429 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" podStartSLOduration=1.867761104 podStartE2EDuration="2.347413791s" totalImagesPullingTime="479.652687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:57.346604549 +0000 UTC m=+4695.044718314" watchObservedRunningTime="2026-10-31 01:37:57.347413791 +0000 UTC m=+4695.045527556" Oct 31 01:38:02.874938 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:02.874902 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:03.182160 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:38:03.182078 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:38:03.297562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.297521 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs"] Oct 31 01:38:03.302390 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.302361 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.319512 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.319445 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1c206c91-openshift-default-dockercfg-bhhxf\"" Oct 31 01:38:03.320153 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.320128 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:38:03.331618 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.331585 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs"] Oct 31 01:38:03.407829 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.407793 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:03.416759 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.416712 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-envoy\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.416941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.416773 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hg9b\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-kube-api-access-8hg9b\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.416941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.416861 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-podinfo\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.416941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.416896 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-credential-socket\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.416941 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.416924 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-certs\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.417147 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.416946 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-data\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.417147 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.417001 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-socket\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.417147 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.417027 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-token\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.417284 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.417178 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istiod-ca-cert\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.518547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.518404 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istiod-ca-cert\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.518740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.518542 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-envoy\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.518740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.518605 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hg9b\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-kube-api-access-8hg9b\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.518740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.518645 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-podinfo\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.518740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.518668 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-credential-socket\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.518740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.518701 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-certs\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.518740 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.518722 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-data\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.519127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.518790 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-socket\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.519127 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.518822 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-token\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.519287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.519261 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-certs\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.519378 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.519355 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-credential-socket\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.519419 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.519377 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-socket\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.519498 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.519433 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istiod-ca-cert\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.519641 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.519619 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-data\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.521281 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.521255 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-envoy\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.521724 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.521704 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-podinfo\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.544975 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.544938 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hg9b\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-kube-api-access-8hg9b\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.550538 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.550509 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-token\") pod \"e2e-ait-1c206c91-openshift-default-697d6f596c-272rs\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.557628 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.557598 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-749d798f87-xfk49"] Oct 31 01:38:03.562429 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.562403 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.584259 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.584225 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749d798f87-xfk49"] Oct 31 01:38:03.585549 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.585510 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:03.617876 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.617836 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:03.620252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.620225 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e659de18-3e67-482d-a05e-07d732064fea-maas-api-tls\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.620394 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.620270 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e659de18-3e67-482d-a05e-07d732064fea-openshift-service-ca\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.620562 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.620549 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e659de18-3e67-482d-a05e-07d732064fea-metrics-certs\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.620632 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.620576 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mqhp\" (UniqueName: \"kubernetes.io/projected/e659de18-3e67-482d-a05e-07d732064fea-kube-api-access-7mqhp\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.721750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.721713 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e659de18-3e67-482d-a05e-07d732064fea-maas-api-tls\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.721956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.721766 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e659de18-3e67-482d-a05e-07d732064fea-openshift-service-ca\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.721956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.721835 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e659de18-3e67-482d-a05e-07d732064fea-metrics-certs\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.721956 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.721866 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mqhp\" (UniqueName: \"kubernetes.io/projected/e659de18-3e67-482d-a05e-07d732064fea-kube-api-access-7mqhp\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.722521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.722492 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e659de18-3e67-482d-a05e-07d732064fea-openshift-service-ca\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.724532 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.724507 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e659de18-3e67-482d-a05e-07d732064fea-metrics-certs\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.724654 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.724600 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e659de18-3e67-482d-a05e-07d732064fea-maas-api-tls\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.743324 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.743285 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mqhp\" (UniqueName: \"kubernetes.io/projected/e659de18-3e67-482d-a05e-07d732064fea-kube-api-access-7mqhp\") pod \"maas-api-749d798f87-xfk49\" (UID: \"e659de18-3e67-482d-a05e-07d732064fea\") " pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.876147 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.876109 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:03.947595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.947558 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs"] Oct 31 01:38:03.954938 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:38:03.954891 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode171c5fe_b84b_46b6_bccd_4b9ce80ff08f.slice/crio-a7e1f60f68688eca079164161f1f9765c888be4dfea350e62e356aa529ed2b61 WatchSource:0}: Error finding container a7e1f60f68688eca079164161f1f9765c888be4dfea350e62e356aa529ed2b61: Status 404 returned error can't find the container with id a7e1f60f68688eca079164161f1f9765c888be4dfea350e62e356aa529ed2b61 Oct 31 01:38:03.958070 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.958030 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:38:03.958185 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.958118 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:38:03.958185 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:03.958162 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:38:04.178331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:04.178298 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749d798f87-xfk49"] Oct 31 01:38:04.180397 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:38:04.180366 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode659de18_3e67_482d_a05e_07d732064fea.slice/crio-c717ec3982de967ba96ca290488802ebd5304b554ecc9c7b4f008298a03b7d6f WatchSource:0}: Error finding container c717ec3982de967ba96ca290488802ebd5304b554ecc9c7b4f008298a03b7d6f: Status 404 returned error can't find the container with id c717ec3982de967ba96ca290488802ebd5304b554ecc9c7b4f008298a03b7d6f Oct 31 01:38:04.350094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:04.350049 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" event={"ID":"e659de18-3e67-482d-a05e-07d732064fea","Type":"ContainerStarted","Data":"c717ec3982de967ba96ca290488802ebd5304b554ecc9c7b4f008298a03b7d6f"} Oct 31 01:38:04.351904 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:04.351875 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" event={"ID":"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f","Type":"ContainerStarted","Data":"612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba"} Oct 31 01:38:04.352058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:04.351927 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" event={"ID":"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f","Type":"ContainerStarted","Data":"a7e1f60f68688eca079164161f1f9765c888be4dfea350e62e356aa529ed2b61"} Oct 31 01:38:04.394221 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:04.394164 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" podStartSLOduration=1.394149975 podStartE2EDuration="1.394149975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:04.391051167 +0000 UTC m=+4702.089164944" watchObservedRunningTime="2026-10-31 01:38:04.394149975 +0000 UTC m=+4702.092263740" Oct 31 01:38:04.618841 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:04.618817 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:05.359382 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:05.359344 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" event={"ID":"e659de18-3e67-482d-a05e-07d732064fea","Type":"ContainerStarted","Data":"35326637addc18f068da814c20c0d2ff5cf9c6851bbc53f78d2706a44694cd45"} Oct 31 01:38:05.359887 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:05.359666 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:05.398179 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:05.398122 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" podStartSLOduration=2.025753417 podStartE2EDuration="2.398106119s" totalImagesPullingTime="372.352702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:05.396374688 +0000 UTC m=+4703.094488453" watchObservedRunningTime="2026-10-31 01:38:05.398106119 +0000 UTC m=+4703.096219884" Oct 31 01:38:05.624362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:05.624286 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:06.165697 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:06.165657 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:06.327930 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:06.327885 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:06.365354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:06.365327 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:06.366266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:06.366248 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:09.216273 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:09.216228 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:09.392371 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:09.392316 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:09.535720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:09.535608 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:10.899441 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:10.899396 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:11.370715 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.370685 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-749d798f87-xfk49" Oct 31 01:38:11.440540 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.440437 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq"] Oct 31 01:38:11.440805 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.440780 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" podUID="0a449cbc-1662-4a50-adbf-3d5d0a18d86a" containerName="maas-api" containerID="cri-o://db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa" gracePeriod=30 Oct 31 01:38:11.699109 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.699083 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:38:11.817808 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.817773 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-maas-api-tls\") pod \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " Oct 31 01:38:11.818026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.817821 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-openshift-service-ca\") pod \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " Oct 31 01:38:11.818026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.817863 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-metrics-certs\") pod \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " Oct 31 01:38:11.818026 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.817930 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-kube-api-access-q57gp\" (UniqueName: \"kubernetes.io/projected/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-kube-api-access-q57gp\") pod \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\" (UID: \"0a449cbc-1662-4a50-adbf-3d5d0a18d86a\") " Oct 31 01:38:11.818296 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.818268 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-openshift-service-ca" pod "0a449cbc-1662-4a50-adbf-3d5d0a18d86a" (UID: "0a449cbc-1662-4a50-adbf-3d5d0a18d86a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:11.820021 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.819975 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-metrics-certs" pod "0a449cbc-1662-4a50-adbf-3d5d0a18d86a" (UID: "0a449cbc-1662-4a50-adbf-3d5d0a18d86a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:38:11.820142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.820119 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-maas-api-tls" pod "0a449cbc-1662-4a50-adbf-3d5d0a18d86a" (UID: "0a449cbc-1662-4a50-adbf-3d5d0a18d86a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:38:11.820574 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.820552 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-kube-api-access-q57gp" pod "0a449cbc-1662-4a50-adbf-3d5d0a18d86a" (UID: "0a449cbc-1662-4a50-adbf-3d5d0a18d86a"). InnerVolumeSpecName "kube-api-access-q57gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:11.918862 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.918825 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:11.918862 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.918857 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:11.918862 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.918868 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:11.919340 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:11.918877 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q57gp\" (UniqueName: \"kubernetes.io/projected/0a449cbc-1662-4a50-adbf-3d5d0a18d86a-kube-api-access-q57gp\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:12.400670 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.400632 5978 generic.go:356] "Generic (PLEG): container finished" podID="0a449cbc-1662-4a50-adbf-3d5d0a18d86a" containerID="db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa" exitCode=0 Oct 31 01:38:12.400875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.400686 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" event={"ID":"0a449cbc-1662-4a50-adbf-3d5d0a18d86a","Type":"ContainerDied","Data":"db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa"} Oct 31 01:38:12.400875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.400690 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" Oct 31 01:38:12.400875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.400728 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq" event={"ID":"0a449cbc-1662-4a50-adbf-3d5d0a18d86a","Type":"ContainerDied","Data":"0c664c4fd65d3c62c0956de0a55b7020057bc7e8ad365cd779ac43655ca255ca"} Oct 31 01:38:12.400875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.400748 5978 scope.go:122] "RemoveContainer" containerID="db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa" Oct 31 01:38:12.411268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.411246 5978 scope.go:122] "RemoveContainer" containerID="db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa" Oct 31 01:38:12.411551 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:38:12.411526 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa\": container with ID starting with db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa not found: ID does not exist" containerID="db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa" Oct 31 01:38:12.411637 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.411561 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa"} err="failed to get container status \"db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa\": rpc error: code = NotFound desc = could not find container \"db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa\": container with ID starting with db0db15fd6e13f373da6302990da555eddfc720198b86700731c3566d9c2b1fa not found: ID does not exist" Oct 31 01:38:12.445893 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.445850 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq"] Oct 31 01:38:12.455356 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.455328 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fb4ff9887-4wgbq"] Oct 31 01:38:12.485134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:12.485097 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:14.102966 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:14.102914 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:14.440603 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:14.440521 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a449cbc-1662-4a50-adbf-3d5d0a18d86a" path="/var/lib/kubelet/pods/0a449cbc-1662-4a50-adbf-3d5d0a18d86a/volumes" Oct 31 01:38:15.746141 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:15.746105 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:17.437398 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:17.437361 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:18.988609 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:18.988572 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:20.613250 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:20.613210 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:22.341507 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:22.341464 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:22.408848 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:22.408663 5978 patch_prober.go:29] interesting pod/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:38:22.408848 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:22.408756 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:38:22.486231 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:22.486196 5978 generic.go:356] "Generic (PLEG): container finished" podID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerID="612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba" exitCode=0 Oct 31 01:38:22.486423 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:22.486272 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" event={"ID":"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f","Type":"ContainerDied","Data":"612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba"} Oct 31 01:38:22.486725 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:22.486709 5978 scope.go:122] "RemoveContainer" containerID="612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba" Oct 31 01:38:22.487991 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:22.487964 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:38:22.488053 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:22.488030 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:38:22.488094 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:22.488061 5978 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 01:38:23.494095 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:23.494057 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" event={"ID":"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f","Type":"ContainerStarted","Data":"0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b"} Oct 31 01:38:23.632478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:23.627892 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:23.632478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:23.628023 5978 patch_prober.go:29] interesting pod/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:38:23.632478 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:23.628066 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:38:24.027393 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:24.027310 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:24.632521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:24.631976 5978 patch_prober.go:29] interesting pod/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:38:24.632521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:24.632045 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:38:25.589551 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:25.589505 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:25.621978 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:25.621935 5978 patch_prober.go:29] interesting pod/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:38:25.623035 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:25.623002 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:38:26.629476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:26.618931 5978 patch_prober.go:29] interesting pod/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:38:26.629476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:26.619002 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:38:27.094084 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:27.094047 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:27.629477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:27.628900 5978 patch_prober.go:29] interesting pod/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:38:27.629477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:27.628969 5978 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:38:28.654095 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:28.654048 5978 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:28.655008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:28.654986 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:28.655669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:28.655607 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:28.676857 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:28.676822 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:30.286951 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:30.286907 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:31.901929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:31.901882 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:32.876675 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:32.876641 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:33.030595 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:33.030547 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:33.181883 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:38:33.181797 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:38:33.483266 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:33.483173 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:34.811601 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:34.811558 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:35.275444 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:35.275406 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:35.421081 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:35.421046 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:38.592637 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:38.592598 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs"] Oct 31 01:38:38.593146 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:38.592815 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" containerID="cri-o://0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b" gracePeriod=30 Oct 31 01:38:38.661362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:38.661317 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:39.975177 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:39.975144 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:40.431183 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:40.431148 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:40.747720 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:40.747626 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:43.656550 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.656513 5978 patch_prober.go:29] interesting pod/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:38:43.657033 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.656566 5978 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:38:43.853216 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.853184 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:43.910145 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910060 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istiod-ca-cert\") pod \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " Oct 31 01:38:43.910145 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910096 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-data\") pod \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " Oct 31 01:38:43.910145 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910142 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-envoy\") pod \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " Oct 31 01:38:43.910474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910182 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-credential-socket\") pod \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " Oct 31 01:38:43.910474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910248 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-token\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-token\") pod \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " Oct 31 01:38:43.910474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910303 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-socket\") pod \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " Oct 31 01:38:43.910474 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910397 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-certs\") pod \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " Oct 31 01:38:43.910744 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910486 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-kube-api-access-8hg9b\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-kube-api-access-8hg9b\") pod \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " Oct 31 01:38:43.910744 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910536 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-podinfo\") pod \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\" (UID: \"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f\") " Oct 31 01:38:43.910744 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910691 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istiod-ca-cert" pod "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" (UID: "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:43.911009 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.910983 5978 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istiod-ca-cert\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:43.911669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.911251 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-certs" pod "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" (UID: "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:38:43.911669 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.911412 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-socket" pod "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" (UID: "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:38:43.911911 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.911890 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-data" pod "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" (UID: "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:38:43.912364 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.912342 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-credential-socket" pod "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" (UID: "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:38:43.913433 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.913407 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-token" pod "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" (UID: "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:43.914250 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.914229 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-kube-api-access-8hg9b" pod "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" (UID: "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f"). InnerVolumeSpecName "kube-api-access-8hg9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:43.914890 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.914855 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-podinfo" pod "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" (UID: "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:38:43.916020 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:43.915994 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-envoy" pod "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" (UID: "e171c5fe-b84b-46b6-bccd-4b9ce80ff08f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:38:44.012057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.012021 5978 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-envoy\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:44.012057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.012054 5978 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-credential-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:44.012057 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.012063 5978 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-token\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:44.012318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.012073 5978 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-socket\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:44.012318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.012081 5978 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-workload-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:44.012318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.012090 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hg9b\" (UniqueName: \"kubernetes.io/projected/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-kube-api-access-8hg9b\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:44.012318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.012099 5978 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-podinfo\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:44.012318 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.012108 5978 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f-istio-data\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:38:44.100575 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.100536 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:44.280719 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.280601 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:44.643826 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.643792 5978 generic.go:356] "Generic (PLEG): container finished" podID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerID="0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b" exitCode=0 Oct 31 01:38:44.644022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.643845 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" Oct 31 01:38:44.644022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.643872 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" event={"ID":"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f","Type":"ContainerDied","Data":"0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b"} Oct 31 01:38:44.644022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.643933 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs" event={"ID":"e171c5fe-b84b-46b6-bccd-4b9ce80ff08f","Type":"ContainerDied","Data":"a7e1f60f68688eca079164161f1f9765c888be4dfea350e62e356aa529ed2b61"} Oct 31 01:38:44.644022 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.643964 5978 scope.go:122] "RemoveContainer" containerID="0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b" Oct 31 01:38:44.655863 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.655821 5978 scope.go:122] "RemoveContainer" containerID="612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba" Oct 31 01:38:44.664440 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.664422 5978 scope.go:122] "RemoveContainer" containerID="0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b" Oct 31 01:38:44.664762 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:38:44.664738 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b\": container with ID starting with 0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b not found: ID does not exist" containerID="0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b" Oct 31 01:38:44.664812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.664769 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b"} err="failed to get container status \"0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b\": rpc error: code = NotFound desc = could not find container \"0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b\": container with ID starting with 0d508cb2b369edb2cff500ebd21c6f681a5980aed7b505d8e301eb4c69dddc5b not found: ID does not exist" Oct 31 01:38:44.664812 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.664788 5978 scope.go:122] "RemoveContainer" containerID="612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba" Oct 31 01:38:44.665015 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:38:44.665000 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba\": container with ID starting with 612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba not found: ID does not exist" containerID="612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba" Oct 31 01:38:44.665059 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.665018 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba"} err="failed to get container status \"612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba\": rpc error: code = NotFound desc = could not find container \"612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba\": container with ID starting with 612ae8cb9b2a43dcbcb7fe38c59d90c8a014e46dce4f7f64e2873429280342ba not found: ID does not exist" Oct 31 01:38:44.709130 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.709091 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs"] Oct 31 01:38:44.718859 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:44.718831 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1c206c91-openshift-default-697d6f596c-272rs"] Oct 31 01:38:45.431521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.431486 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m"] Oct 31 01:38:45.431968 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.431956 5978 cpu_manager.go:414] "containerMap: removing container" podUID="0a449cbc-1662-4a50-adbf-3d5d0a18d86a" containerName="maas-api" Oct 31 01:38:45.432016 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.431970 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a449cbc-1662-4a50-adbf-3d5d0a18d86a" containerName="maas-api" Oct 31 01:38:45.432016 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.431984 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" Oct 31 01:38:45.432016 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.431989 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" Oct 31 01:38:45.432016 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.432002 5978 cpu_manager.go:414] "containerMap: removing container" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" Oct 31 01:38:45.432016 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.432007 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" Oct 31 01:38:45.432226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.432066 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" Oct 31 01:38:45.432226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.432074 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" containerName="istio-proxy" Oct 31 01:38:45.432226 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.432088 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a449cbc-1662-4a50-adbf-3d5d0a18d86a" containerName="maas-api" Oct 31 01:38:45.436487 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.436466 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.454630 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.454600 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-aebdfd-serving-cert\"" Oct 31 01:38:45.462447 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.462413 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m"] Oct 31 01:38:45.531064 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.531021 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.531261 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.531083 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2bd\" (UniqueName: \"kubernetes.io/projected/dfd340fc-56b7-49c4-8d42-b1bf09125773-kube-api-access-jd2bd\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.531261 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.531236 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd340fc-56b7-49c4-8d42-b1bf09125773-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.531342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.531290 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-metrics-certs\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.549386 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.549350 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:45.632753 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.632719 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.632983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.632759 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2bd\" (UniqueName: \"kubernetes.io/projected/dfd340fc-56b7-49c4-8d42-b1bf09125773-kube-api-access-jd2bd\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.632983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.632822 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd340fc-56b7-49c4-8d42-b1bf09125773-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.632983 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.632859 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-metrics-certs\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.633723 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.633690 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd340fc-56b7-49c4-8d42-b1bf09125773-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.635399 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.635371 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.635560 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.635415 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-metrics-certs\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.654186 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.654156 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2bd\" (UniqueName: \"kubernetes.io/projected/dfd340fc-56b7-49c4-8d42-b1bf09125773-kube-api-access-jd2bd\") pod \"maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.712527 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.712409 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:45.751136 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.751094 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:45.874900 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:45.874853 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:46.037067 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:46.036966 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:46.067791 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:46.067765 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m"] Oct 31 01:38:46.071330 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:38:46.071280 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd340fc_56b7_49c4_8d42_b1bf09125773.slice/crio-12d933377be10073c8e2f9106307b46d1c55a26e8e5896f6f68c7a7ffa7001f3 WatchSource:0}: Error finding container 12d933377be10073c8e2f9106307b46d1c55a26e8e5896f6f68c7a7ffa7001f3: Status 404 returned error can't find the container with id 12d933377be10073c8e2f9106307b46d1c55a26e8e5896f6f68c7a7ffa7001f3 Oct 31 01:38:46.423899 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:46.423864 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e171c5fe-b84b-46b6-bccd-4b9ce80ff08f" path="/var/lib/kubelet/pods/e171c5fe-b84b-46b6-bccd-4b9ce80ff08f/volumes" Oct 31 01:38:46.657302 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:46.657266 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" event={"ID":"dfd340fc-56b7-49c4-8d42-b1bf09125773","Type":"ContainerStarted","Data":"50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92"} Oct 31 01:38:46.657532 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:46.657323 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:46.657532 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:46.657343 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" event={"ID":"dfd340fc-56b7-49c4-8d42-b1bf09125773","Type":"ContainerStarted","Data":"12d933377be10073c8e2f9106307b46d1c55a26e8e5896f6f68c7a7ffa7001f3"} Oct 31 01:38:46.696333 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:46.696215 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" podStartSLOduration=1.34143836 podStartE2EDuration="1.696197038s" totalImagesPullingTime="354.758678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:46.693931095 +0000 UTC m=+4744.392044872" watchObservedRunningTime="2026-10-31 01:38:46.696197038 +0000 UTC m=+4744.394310807" Oct 31 01:38:47.300946 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:47.300911 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:48.885348 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:48.885317 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:49.577980 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:49.577946 5978 scope.go:122] "RemoveContainer" containerID="eab980c99e1257b49feb2040e2503cc70b63d00f8a1130921870ac3ea9064d0c" Oct 31 01:38:50.538011 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:50.537974 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:52.320046 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:52.320010 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:52.668844 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:52.668771 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:38:52.746232 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:52.746196 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:54.093577 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:54.093496 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:55.969349 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:55.969311 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:57.496589 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:57.496550 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:58.984825 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:58.984782 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:59.125680 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:38:59.125645 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:00.820686 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:00.820644 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:02.599814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:02.599773 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:03.181795 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:39:03.181757 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:39:04.228303 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:04.228222 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:05.882840 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:05.882795 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:07.392143 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:07.392101 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:08.904706 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:08.904667 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:10.392817 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:10.392774 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:11.896302 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:11.896258 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:13.502503 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:13.502447 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:15.068958 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:15.068921 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:16.700773 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:16.700730 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:18.404112 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:18.404076 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:19.992204 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:19.992159 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:21.537730 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:21.537691 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:23.108435 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:23.108392 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:24.030790 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:24.030700 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775"] Oct 31 01:39:24.042583 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:24.042540 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cs775"] Oct 31 01:39:24.423901 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:24.423865 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3893383-addc-4ec8-9d4d-0ccf412b1857" path="/var/lib/kubelet/pods/e3893383-addc-4ec8-9d4d-0ccf412b1857/volumes" Oct 31 01:39:24.675282 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:24.675201 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:26.089903 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:26.089870 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:26.229901 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:26.229869 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:27.778311 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:27.778271 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:29.373028 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:29.372991 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:30.902019 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:30.901982 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:32.499394 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:32.499353 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:33.180660 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:39:33.180619 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:39:34.076331 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:34.076290 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:35.579432 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:35.579397 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:37.149304 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:37.149271 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:38.690743 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:38.690698 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:40.202352 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:40.202315 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:41.805222 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:41.805186 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:43.294515 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:43.294472 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:44.890497 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:44.890443 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:45.029512 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:45.029407 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk"] Oct 31 01:39:45.041424 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:45.041394 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-m6qvk"] Oct 31 01:39:46.277088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:46.277049 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:46.423526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:46.423489 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:46.424035 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:46.424013 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d583e6a7-7b53-40a0-bc77-cc6e72cf613a" path="/var/lib/kubelet/pods/d583e6a7-7b53-40a0-bc77-cc6e72cf613a/volumes" Oct 31 01:39:46.594886 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:46.594850 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:48.103907 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:48.103867 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:49.842058 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:49.842017 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:50.814850 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:50.814822 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:39:50.827111 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:50.827085 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sgkkv_064705a9-54fd-4528-ae88-1cb4596c5b6e/ovn-acl-logging/0.log" Oct 31 01:39:51.426405 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:51.426351 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:52.855802 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:52.855763 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:53.056030 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:53.055997 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:54.491855 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:54.491813 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:56.013521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:56.013484 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:57.677041 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:57.677003 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:59.240476 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:39:59.240414 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:00.788357 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:00.788321 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:02.388721 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:02.388677 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:03.181172 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:40:03.181130 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:40:03.994512 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:03.994483 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:05.586409 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:05.586372 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:07.190644 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:07.190606 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:08.800140 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:08.800100 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:10.399384 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:10.399348 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:11.910967 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:11.910922 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:13.487937 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:13.487891 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:15.080767 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:15.080729 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:16.584531 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:16.584498 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:18.182774 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:18.182734 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:19.822691 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:19.822647 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:21.500844 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:21.500811 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:23.101400 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:23.101362 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:24.692481 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:24.692423 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:26.292000 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:26.291962 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:27.896172 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:27.896133 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:29.389978 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:29.389940 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:29.964641 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:29.964600 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:30.165008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:30.164965 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:33.181290 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:40:33.181251 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:40:35.644580 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:35.644542 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m"] Oct 31 01:40:35.645006 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:35.644756 5978 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" podUID="dfd340fc-56b7-49c4-8d42-b1bf09125773" containerName="maas-api" containerID="cri-o://50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92" gracePeriod=30 Oct 31 01:40:35.762508 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:35.762473 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:35.919563 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:35.919532 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:40:36.008095 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.008059 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-maas-api-tls\") pod \"dfd340fc-56b7-49c4-8d42-b1bf09125773\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " Oct 31 01:40:36.008289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.008115 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd340fc-56b7-49c4-8d42-b1bf09125773-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd340fc-56b7-49c4-8d42-b1bf09125773-openshift-service-ca\") pod \"dfd340fc-56b7-49c4-8d42-b1bf09125773\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " Oct 31 01:40:36.008289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.008143 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd340fc-56b7-49c4-8d42-b1bf09125773-kube-api-access-jd2bd\" (UniqueName: \"kubernetes.io/projected/dfd340fc-56b7-49c4-8d42-b1bf09125773-kube-api-access-jd2bd\") pod \"dfd340fc-56b7-49c4-8d42-b1bf09125773\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " Oct 31 01:40:36.008289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.008178 5978 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-metrics-certs\") pod \"dfd340fc-56b7-49c4-8d42-b1bf09125773\" (UID: \"dfd340fc-56b7-49c4-8d42-b1bf09125773\") " Oct 31 01:40:36.008628 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.008599 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd340fc-56b7-49c4-8d42-b1bf09125773-openshift-service-ca" pod "dfd340fc-56b7-49c4-8d42-b1bf09125773" (UID: "dfd340fc-56b7-49c4-8d42-b1bf09125773"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:36.010215 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.010190 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-maas-api-tls" pod "dfd340fc-56b7-49c4-8d42-b1bf09125773" (UID: "dfd340fc-56b7-49c4-8d42-b1bf09125773"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:36.010342 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.010305 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd340fc-56b7-49c4-8d42-b1bf09125773-kube-api-access-jd2bd" pod "dfd340fc-56b7-49c4-8d42-b1bf09125773" (UID: "dfd340fc-56b7-49c4-8d42-b1bf09125773"). InnerVolumeSpecName "kube-api-access-jd2bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:36.010405 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.010336 5978 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-metrics-certs" pod "dfd340fc-56b7-49c4-8d42-b1bf09125773" (UID: "dfd340fc-56b7-49c4-8d42-b1bf09125773"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:36.109392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.109352 5978 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-maas-api-tls\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:40:36.109392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.109385 5978 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd340fc-56b7-49c4-8d42-b1bf09125773-openshift-service-ca\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:40:36.109392 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.109396 5978 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2bd\" (UniqueName: \"kubernetes.io/projected/dfd340fc-56b7-49c4-8d42-b1bf09125773-kube-api-access-jd2bd\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:40:36.109782 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.109406 5978 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd340fc-56b7-49c4-8d42-b1bf09125773-metrics-certs\") on node \"ip-10-0-129-199.ec2.internal\" DevicePath \"\"" Oct 31 01:40:36.235552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.235467 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:36.264197 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.264163 5978 generic.go:356] "Generic (PLEG): container finished" podID="dfd340fc-56b7-49c4-8d42-b1bf09125773" containerID="50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92" exitCode=0 Oct 31 01:40:36.264377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.264212 5978 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" Oct 31 01:40:36.264377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.264221 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" event={"ID":"dfd340fc-56b7-49c4-8d42-b1bf09125773","Type":"ContainerDied","Data":"50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92"} Oct 31 01:40:36.264377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.264266 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m" event={"ID":"dfd340fc-56b7-49c4-8d42-b1bf09125773","Type":"ContainerDied","Data":"12d933377be10073c8e2f9106307b46d1c55a26e8e5896f6f68c7a7ffa7001f3"} Oct 31 01:40:36.264377 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.264287 5978 scope.go:122] "RemoveContainer" containerID="50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92" Oct 31 01:40:36.274372 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.274351 5978 scope.go:122] "RemoveContainer" containerID="50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92" Oct 31 01:40:36.274697 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:40:36.274679 5978 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92\": container with ID starting with 50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92 not found: ID does not exist" containerID="50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92" Oct 31 01:40:36.274802 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.274707 5978 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92"} err="failed to get container status \"50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92\": rpc error: code = NotFound desc = could not find container \"50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92\": container with ID starting with 50a5be8a06a1ffc24f929a3d560435041482b9d594e7b143e32887ed11ac0a92 not found: ID does not exist" Oct 31 01:40:36.316272 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.316237 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m"] Oct 31 01:40:36.320408 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.320376 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-aebdfd-68fd7dd6f4-rff8m"] Oct 31 01:40:36.378929 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.378887 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:36.424233 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.424193 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd340fc-56b7-49c4-8d42-b1bf09125773" path="/var/lib/kubelet/pods/dfd340fc-56b7-49c4-8d42-b1bf09125773/volumes" Oct 31 01:40:36.887305 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:36.887270 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:43.127855 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:43.127818 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:43.274972 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:43.274928 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:43.544042 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:43.543958 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:49.620766 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:49.620730 5978 scope.go:122] "RemoveContainer" containerID="693a6d58e7ea042ad9473e241786d71d180428f1fb0b96aedf82d1cb076986b9" Oct 31 01:40:49.629909 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:49.629886 5978 scope.go:122] "RemoveContainer" containerID="26b4d3a05e8bfe0614f66a6fc62c452e48f31a9cc3dba4ad29118e567eb3f11b" Oct 31 01:40:52.991589 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:40:52.991552 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:03.181237 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:41:03.181150 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:41:33.181916 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:41:33.181874 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:41:40.676988 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:40.676959 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c6f985896-klgtr_ee5bd20d-24df-423b-928c-872d211c46bd/authorino/0.log" Oct 31 01:41:52.473947 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:52.473910 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-g5g96_911b72ad-f153-41d5-86e2-05bcc281c020/manager/0.log" Oct 31 01:41:53.040410 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:53.040373 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:53.709008 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:53.708972 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6tv7g_097ad6a1-77bb-405a-8fb5-838805f787c4/proxy/0.log" Oct 31 01:41:54.241585 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:54.241536 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:54.680700 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:54.680666 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-749d798f87-xfk49_e659de18-3e67-482d-a05e-07d732064fea/maas-api/0.log" Oct 31 01:41:55.067853 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:55.067822 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-03496072-hl7hb_a52333b1-a32e-487d-85a6-76a4e130dc5a/revoke-keys/0.log" Oct 31 01:41:55.264362 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:55.264331 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-vqq8b_c900896d-c6eb-43cd-befd-93a5c5c758e9/revoke-keys/0.log" Oct 31 01:41:55.456658 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:55.456578 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-5gljx_b0f9c96b-dbb2-455d-bbb3-83de84ea7932/revoke-keys/0.log" Oct 31 01:41:57.548431 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:41:57.548398 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c6f985896-klgtr_ee5bd20d-24df-423b-928c-872d211c46bd/authorino/0.log" Oct 31 01:42:00.271079 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:00.271042 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:00.282289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:00.282261 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94bcdf98f-xh2jv_fbcb321a-95e1-414c-bafc-c819681071a1/kube-auth-proxy/0.log" Oct 31 01:42:00.472134 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:00.472104 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6bxb6_694e2b84-5e34-4d1e-bf99-16b261a95272/istio-proxy/0.log" Oct 31 01:42:01.264614 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:01.264582 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-ldf84_95964f90-7a0d-4fe8-97aa-6f10cf619a81/istio-proxy/0.log" Oct 31 01:42:01.664822 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:01.664787 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-nqx6q_b6de6a36-1b82-49ef-aa57-c476ced0542f/istio-proxy/0.log" Oct 31 01:42:01.870164 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:01.870136 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qq9v6_b4643839-1a91-40bd-b0f2-d335540a1ad5/istio-proxy/0.log" Oct 31 01:42:02.868142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:02.868110 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:42:03.181126 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:42:03.181038 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:42:04.299188 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:04.299153 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-cckrj_51fb742e-ddac-4178-8f67-38f62e8d3f71/main/0.log" Oct 31 01:42:04.510517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:04.510481 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-kl9w5_0dcb20e6-3314-47bb-983e-40d23e003f6d/main/0.log" Oct 31 01:42:05.526298 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:05.526269 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9_847df7f7-0054-4cc9-a296-e64c9f10dd2d/main/0.log" Oct 31 01:42:06.029365 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:06.029329 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx"] Oct 31 01:42:06.041385 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:06.041346 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5gljx"] Oct 31 01:42:06.423906 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:06.423874 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f9c96b-dbb2-455d-bbb3-83de84ea7932" path="/var/lib/kubelet/pods/b0f9c96b-dbb2-455d-bbb3-83de84ea7932/volumes" Oct 31 01:42:06.472873 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:06.472841 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c6f985896-klgtr_ee5bd20d-24df-423b-928c-872d211c46bd/authorino/0.log" Oct 31 01:42:16.228722 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:16.228687 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-g5g96_911b72ad-f153-41d5-86e2-05bcc281c020/manager/0.log" Oct 31 01:42:17.364772 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:17.364736 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6tv7g_097ad6a1-77bb-405a-8fb5-838805f787c4/proxy/0.log" Oct 31 01:42:18.136685 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:18.136646 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-749d798f87-xfk49_e659de18-3e67-482d-a05e-07d732064fea/maas-api/0.log" Oct 31 01:42:18.529196 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:18.529116 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-03496072-hl7hb_a52333b1-a32e-487d-85a6-76a4e130dc5a/revoke-keys/0.log" Oct 31 01:42:18.721363 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:18.721325 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-vqq8b_c900896d-c6eb-43cd-befd-93a5c5c758e9/revoke-keys/0.log" Oct 31 01:42:20.434088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:20.434039 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c6f985896-klgtr_ee5bd20d-24df-423b-928c-872d211c46bd/authorino/0.log" Oct 31 01:42:22.163558 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:22.163523 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:22.878916 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:22.878881 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94bcdf98f-xh2jv_fbcb321a-95e1-414c-bafc-c819681071a1/kube-auth-proxy/0.log" Oct 31 01:42:23.067602 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:23.067571 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6bxb6_694e2b84-5e34-4d1e-bf99-16b261a95272/istio-proxy/0.log" Oct 31 01:42:23.874935 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:23.874713 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-ldf84_95964f90-7a0d-4fe8-97aa-6f10cf619a81/istio-proxy/0.log" Oct 31 01:42:24.030297 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:24.030259 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b"] Oct 31 01:42:24.042432 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:24.042395 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vqq8b"] Oct 31 01:42:24.282982 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:24.282899 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-nqx6q_b6de6a36-1b82-49ef-aa57-c476ced0542f/istio-proxy/0.log" Oct 31 01:42:24.424130 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:24.424083 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c900896d-c6eb-43cd-befd-93a5c5c758e9" path="/var/lib/kubelet/pods/c900896d-c6eb-43cd-befd-93a5c5c758e9/volumes" Oct 31 01:42:24.487520 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:24.487490 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qq9v6_b4643839-1a91-40bd-b0f2-d335540a1ad5/istio-proxy/0.log" Oct 31 01:42:25.418124 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:25.418092 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:42:26.352182 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:26.352147 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-cckrj_51fb742e-ddac-4178-8f67-38f62e8d3f71/main/0.log" Oct 31 01:42:26.540291 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:26.540257 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-kl9w5_0dcb20e6-3314-47bb-983e-40d23e003f6d/main/0.log" Oct 31 01:42:27.525287 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:27.525259 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-h9nd9_847df7f7-0054-4cc9-a296-e64c9f10dd2d/main/0.log" Oct 31 01:42:33.181539 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:42:33.181498 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:42:47.030195 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:47.030154 5978 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb"] Oct 31 01:42:47.041949 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:47.041914 5978 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03496072-hl7hb"] Oct 31 01:42:48.424013 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:48.423978 5978 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52333b1-a32e-487d-85a6-76a4e130dc5a" path="/var/lib/kubelet/pods/a52333b1-a32e-487d-85a6-76a4e130dc5a/volumes" Oct 31 01:42:53.260027 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:42:53.259981 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:01.399729 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:01.399695 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:43:01.601516 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:01.601490 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:43:03.181552 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:43:03.181510 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:43:33.181397 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:43:33.181356 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:43:44.280332 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.280305 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:43:44.471517 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.471491 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log" Oct 31 01:43:44.839713 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.839679 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9cckr/must-gather-rktdk"] Oct 31 01:43:44.840162 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.840147 5978 cpu_manager.go:414] "containerMap: removing container" podUID="dfd340fc-56b7-49c4-8d42-b1bf09125773" containerName="maas-api" Oct 31 01:43:44.840218 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.840165 5978 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd340fc-56b7-49c4-8d42-b1bf09125773" containerName="maas-api" Oct 31 01:43:44.840258 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.840242 5978 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd340fc-56b7-49c4-8d42-b1bf09125773" containerName="maas-api" Oct 31 01:43:44.843506 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.843488 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cckr/must-gather-rktdk" Oct 31 01:43:44.858501 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.858475 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cckr\"/\"kube-root-ca.crt\"" Oct 31 01:43:44.870289 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.870272 5978 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cckr\"/\"openshift-service-ca.crt\"" Oct 31 01:43:44.870504 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.870486 5978 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9cckr\"/\"default-dockercfg-l8p77\"" Oct 31 01:43:44.873913 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.873887 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:44.876954 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.876934 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/53195d48-c7d6-40cd-a1bf-a69f8acdee67-must-gather-output\") pod \"must-gather-rktdk\" (UID: \"53195d48-c7d6-40cd-a1bf-a69f8acdee67\") " pod="openshift-must-gather-9cckr/must-gather-rktdk" Oct 31 01:43:44.877054 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.877039 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d6dm\" (UniqueName: \"kubernetes.io/projected/53195d48-c7d6-40cd-a1bf-a69f8acdee67-kube-api-access-7d6dm\") pod \"must-gather-rktdk\" (UID: \"53195d48-c7d6-40cd-a1bf-a69f8acdee67\") " pod="openshift-must-gather-9cckr/must-gather-rktdk" Oct 31 01:43:44.885296 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.885273 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cckr/must-gather-rktdk"] Oct 31 01:43:44.978644 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.978607 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/53195d48-c7d6-40cd-a1bf-a69f8acdee67-must-gather-output\") pod \"must-gather-rktdk\" (UID: \"53195d48-c7d6-40cd-a1bf-a69f8acdee67\") " pod="openshift-must-gather-9cckr/must-gather-rktdk" Oct 31 01:43:44.978834 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.978703 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d6dm\" (UniqueName: \"kubernetes.io/projected/53195d48-c7d6-40cd-a1bf-a69f8acdee67-kube-api-access-7d6dm\") pod \"must-gather-rktdk\" (UID: \"53195d48-c7d6-40cd-a1bf-a69f8acdee67\") " pod="openshift-must-gather-9cckr/must-gather-rktdk" Oct 31 01:43:44.978965 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:44.978947 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/53195d48-c7d6-40cd-a1bf-a69f8acdee67-must-gather-output\") pod \"must-gather-rktdk\" (UID: \"53195d48-c7d6-40cd-a1bf-a69f8acdee67\") " pod="openshift-must-gather-9cckr/must-gather-rktdk" Oct 31 01:43:45.008902 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:45.008870 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d6dm\" (UniqueName: \"kubernetes.io/projected/53195d48-c7d6-40cd-a1bf-a69f8acdee67-kube-api-access-7d6dm\") pod \"must-gather-rktdk\" (UID: \"53195d48-c7d6-40cd-a1bf-a69f8acdee67\") " pod="openshift-must-gather-9cckr/must-gather-rktdk" Oct 31 01:43:45.038659 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:45.038615 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:45.158727 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:45.158641 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cckr/must-gather-rktdk" Oct 31 01:43:45.508750 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:45.503057 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cckr/must-gather-rktdk"] Oct 31 01:43:45.513241 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:45.511259 5978 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:43:46.395399 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:46.395363 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cckr/must-gather-rktdk" event={"ID":"53195d48-c7d6-40cd-a1bf-a69f8acdee67","Type":"ContainerStarted","Data":"94ff8edd6d152dcc6d0074bb9574ec1f53e365004276f88e71e1c7d773dbfab2"} Oct 31 01:43:47.406016 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:47.405975 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cckr/must-gather-rktdk" event={"ID":"53195d48-c7d6-40cd-a1bf-a69f8acdee67","Type":"ContainerStarted","Data":"dfb52489471d8de9bcb4aa37d3c5ac78126b4c1b3bca860513fdab48a6e867b4"} Oct 31 01:43:47.406421 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:47.406030 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cckr/must-gather-rktdk" event={"ID":"53195d48-c7d6-40cd-a1bf-a69f8acdee67","Type":"ContainerStarted","Data":"b8ca0332c17a18ce54a62ddb61df430ec45c62ff1e4ae6a8f67a04b4139c337e"} Oct 31 01:43:47.446319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:47.446265 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9cckr/must-gather-rktdk" podStartSLOduration=2.664913098 podStartE2EDuration="3.446249815s" totalImagesPullingTime="781.336717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:47.445867484 +0000 UTC m=+5045.143981253" watchObservedRunningTime="2026-10-31 01:43:47.446249815 +0000 UTC m=+5045.144363577" Oct 31 01:43:49.677335 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:49.677300 5978 scope.go:122] "RemoveContainer" containerID="697d5b45e718603dbbd4ac555c1554502b45ed7ebd1ce01e4626f59dc5b91270" Oct 31 01:43:49.686286 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:49.686254 5978 scope.go:122] "RemoveContainer" containerID="9d191ccbbfcac7336f5803a2816a66b1d2c0ebbdeab2f4e4594afee0ea7c70a8" Oct 31 01:43:49.694577 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:49.694548 5978 scope.go:122] "RemoveContainer" containerID="9d712ddf5edb62a2e5a9321874ca60e301067d474e23e19ac73c68511254304f" Oct 31 01:43:50.076933 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:50.076901 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z4hfm_a56c984b-c9d9-4d3f-86fd-2314f01a6673/global-pull-secret-syncer/0.log" Oct 31 01:43:50.743885 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:50.743844 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" Oct 31 01:43:51.307687 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:51.304405 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-199.ec2.internal_ad1af2674e0c55384c949adc8f5b6f9c/haproxy/0.log" Oct 31 01:43:53.420489 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:53.420051 5978 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bshvl_1f9f082d-9176-491a-8d5e-281ee6665dae/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:59.361319 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:43:59.361285 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c6f985896-klgtr_ee5bd20d-24df-423b-928c-872d211c46bd/authorino/0.log" Oct 31 01:44:00.210835 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.210793 5978 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8"] Oct 31 01:44:00.220354 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.220318 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.249306 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.249275 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8"] Oct 31 01:44:00.370160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.369712 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnk7\" (UniqueName: \"kubernetes.io/projected/2e91d514-64ac-4e3d-8c00-571bbe729d08-kube-api-access-4mnk7\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.370160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.369790 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-lib-modules\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.370160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.369884 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-proc\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.370160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.369986 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-sys\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.370160 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.370019 5978 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-podres\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.471246 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.471149 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-lib-modules\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.471246 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.471240 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-proc\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.471530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.471265 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-lib-modules\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.471530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.471334 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-sys\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.471530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.471364 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-proc\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.471530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.471369 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-podres\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.471530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.471416 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-sys\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.471530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.471424 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e91d514-64ac-4e3d-8c00-571bbe729d08-podres\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.471530 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.471475 5978 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnk7\" (UniqueName: \"kubernetes.io/projected/2e91d514-64ac-4e3d-8c00-571bbe729d08-kube-api-access-4mnk7\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.495198 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.495152 5978 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnk7\" (UniqueName: \"kubernetes.io/projected/2e91d514-64ac-4e3d-8c00-571bbe729d08-kube-api-access-4mnk7\") pod \"perf-node-gather-daemonset-h5tp8\" (UID: \"2e91d514-64ac-4e3d-8c00-571bbe729d08\") " pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.544817 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.544774 5978 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:00.864791 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:00.864489 5978 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8"] Oct 31 01:44:00.865810 ip-10-0-129-199 kubenswrapper[5978]: W1031 01:44:00.865775 5978 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2e91d514_64ac_4e3d_8c00_571bbe729d08.slice/crio-32856394d86a743326ef2a2c178fc1f3ea5bd9f167c2a96770df90438918d302 WatchSource:0}: Error finding container 32856394d86a743326ef2a2c178fc1f3ea5bd9f167c2a96770df90438918d302: Status 404 returned error can't find the container with id 32856394d86a743326ef2a2c178fc1f3ea5bd9f167c2a96770df90438918d302 Oct 31 01:44:01.461770 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:01.461727 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_99e45dfd-1c7d-4ca5-92ae-65e99eef0e30/alertmanager/0.log" Oct 31 01:44:01.545957 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:01.544857 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" event={"ID":"2e91d514-64ac-4e3d-8c00-571bbe729d08","Type":"ContainerStarted","Data":"2498777e167172b1df60161340c6176deb1f31649eb0ae760130388866e75aad"} Oct 31 01:44:01.545957 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:01.544932 5978 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" event={"ID":"2e91d514-64ac-4e3d-8c00-571bbe729d08","Type":"ContainerStarted","Data":"32856394d86a743326ef2a2c178fc1f3ea5bd9f167c2a96770df90438918d302"} Oct 31 01:44:01.545957 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:01.545841 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:01.582477 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:01.582423 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_99e45dfd-1c7d-4ca5-92ae-65e99eef0e30/config-reloader/0.log" Oct 31 01:44:01.588765 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:01.588715 5978 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" podStartSLOduration=1.58870193 podStartE2EDuration="1.58870193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:01.587748627 +0000 UTC m=+5059.285862411" watchObservedRunningTime="2026-10-31 01:44:01.58870193 +0000 UTC m=+5059.286815694" Oct 31 01:44:01.710088 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:01.710056 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_99e45dfd-1c7d-4ca5-92ae-65e99eef0e30/kube-rbac-proxy-web/0.log" Oct 31 01:44:01.835521 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:01.835487 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_99e45dfd-1c7d-4ca5-92ae-65e99eef0e30/kube-rbac-proxy/0.log" Oct 31 01:44:01.978610 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:01.978567 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_99e45dfd-1c7d-4ca5-92ae-65e99eef0e30/kube-rbac-proxy-metric/0.log" Oct 31 01:44:02.100828 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:02.100733 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_99e45dfd-1c7d-4ca5-92ae-65e99eef0e30/prom-label-proxy/0.log" Oct 31 01:44:02.224591 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:02.224563 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_99e45dfd-1c7d-4ca5-92ae-65e99eef0e30/init-config-reloader/0.log" Oct 31 01:44:02.492814 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:02.492730 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mmhbf_364a8716-433d-4181-94ed-f6ccd33a0828/kube-state-metrics/0.log" Oct 31 01:44:02.604367 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:02.604336 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mmhbf_364a8716-433d-4181-94ed-f6ccd33a0828/kube-rbac-proxy-main/0.log" Oct 31 01:44:02.729014 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:02.728983 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mmhbf_364a8716-433d-4181-94ed-f6ccd33a0828/kube-rbac-proxy-self/0.log" Oct 31 01:44:02.859315 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:02.859288 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6cbf58948-7xm9f_0e0bd8aa-5627-4ece-8ef4-97a529ab89ae/metrics-server/0.log" Oct 31 01:44:02.982163 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:02.982131 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-br9bn_a719455f-b7a1-4394-8258-6d84f4c0ccd3/monitoring-plugin/0.log" Oct 31 01:44:03.182677 ip-10-0-129-199 kubenswrapper[5978]: E1031 01:44:03.182598 5978 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qg2xf" containerName="extractor" Oct 31 01:44:03.876824 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:03.876794 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lhrff_465a3984-adbc-4677-b22f-9e9240a770f8/node-exporter/0.log" Oct 31 01:44:03.993970 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:03.993940 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lhrff_465a3984-adbc-4677-b22f-9e9240a770f8/kube-rbac-proxy/0.log" Oct 31 01:44:04.122551 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:04.122518 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lhrff_465a3984-adbc-4677-b22f-9e9240a770f8/init-textfile/0.log" Oct 31 01:44:04.244677 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:04.244605 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-mwvzf_8a38f09d-12b6-481c-bb13-c686226854a6/kube-rbac-proxy-main/0.log" Oct 31 01:44:04.382185 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:04.382154 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-mwvzf_8a38f09d-12b6-481c-bb13-c686226854a6/kube-rbac-proxy-self/0.log" Oct 31 01:44:04.509798 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:04.509713 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-mwvzf_8a38f09d-12b6-481c-bb13-c686226854a6/openshift-state-metrics/0.log" Oct 31 01:44:04.646875 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:04.646845 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_011c406e-96bb-49b6-bdd8-d4f2b36d636e/prometheus/0.log" Oct 31 01:44:04.769190 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:04.769114 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_011c406e-96bb-49b6-bdd8-d4f2b36d636e/config-reloader/0.log" Oct 31 01:44:04.894738 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:04.894707 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_011c406e-96bb-49b6-bdd8-d4f2b36d636e/thanos-sidecar/0.log" Oct 31 01:44:05.023905 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:05.023799 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_011c406e-96bb-49b6-bdd8-d4f2b36d636e/kube-rbac-proxy-web/0.log" Oct 31 01:44:05.176151 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:05.176123 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_011c406e-96bb-49b6-bdd8-d4f2b36d636e/kube-rbac-proxy/0.log" Oct 31 01:44:05.295816 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:05.295726 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_011c406e-96bb-49b6-bdd8-d4f2b36d636e/kube-rbac-proxy-thanos/0.log" Oct 31 01:44:05.429146 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:05.429116 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_011c406e-96bb-49b6-bdd8-d4f2b36d636e/init-config-reloader/0.log" Oct 31 01:44:06.271552 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:06.271519 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6db9fdb49c-5z6zr_2cb59038-b544-4cb1-a2af-ded4bb5d6431/thanos-query/0.log" Oct 31 01:44:06.394526 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:06.394495 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6db9fdb49c-5z6zr_2cb59038-b544-4cb1-a2af-ded4bb5d6431/kube-rbac-proxy-web/0.log" Oct 31 01:44:06.514434 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:06.514409 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6db9fdb49c-5z6zr_2cb59038-b544-4cb1-a2af-ded4bb5d6431/kube-rbac-proxy/0.log" Oct 31 01:44:06.635252 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:06.635217 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6db9fdb49c-5z6zr_2cb59038-b544-4cb1-a2af-ded4bb5d6431/prom-label-proxy/0.log" Oct 31 01:44:06.757564 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:06.756233 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6db9fdb49c-5z6zr_2cb59038-b544-4cb1-a2af-ded4bb5d6431/kube-rbac-proxy-rules/0.log" Oct 31 01:44:06.881547 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:06.881506 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6db9fdb49c-5z6zr_2cb59038-b544-4cb1-a2af-ded4bb5d6431/kube-rbac-proxy-metrics/0.log" Oct 31 01:44:08.569830 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:08.569796 5978 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9cckr/perf-node-gather-daemonset-h5tp8" Oct 31 01:44:10.476029 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:10.475001 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c5bccddfc-p44kw_1bc12e8e-e1be-49ee-8a5c-32026f7e656e/console/0.log" Oct 31 01:44:14.590810 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:14.590774 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nqpmv_d6f8e29e-fb86-4153-b168-0bb1743e44cd/dns/0.log" Oct 31 01:44:14.704268 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:14.704236 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nqpmv_d6f8e29e-fb86-4153-b168-0bb1743e44cd/kube-rbac-proxy/0.log" Oct 31 01:44:15.357661 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:15.357629 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-npncc_47350ce3-ba5e-4393-ac0f-653b41bc28fd/dns-node-resolver/0.log" Oct 31 01:44:16.764234 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:16.764203 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jpxt2_f99f8918-3e3d-456b-be91-db4120b67424/node-ca/0.log" Oct 31 01:44:18.825213 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:18.825178 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-94bcdf98f-xh2jv_fbcb321a-95e1-414c-bafc-c819681071a1/kube-auth-proxy/0.log" Oct 31 01:44:18.908837 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:18.908807 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6bxb6_694e2b84-5e34-4d1e-bf99-16b261a95272/istio-proxy/0.log" Oct 31 01:44:19.187280 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:19.187200 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-ldf84_95964f90-7a0d-4fe8-97aa-6f10cf619a81/istio-proxy/0.log" Oct 31 01:44:19.321796 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:19.321767 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-nqx6q_b6de6a36-1b82-49ef-aa57-c476ced0542f/istio-proxy/0.log" Oct 31 01:44:19.398142 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:19.398096 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qq9v6_b4643839-1a91-40bd-b0f2-d335540a1ad5/istio-proxy/0.log" Oct 31 01:44:19.742588 ip-10-0-129-199 kubenswrapper[5978]: I1031 01:44:19.742544 5978 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7vfd8_33311d84-f2dc-4d32-b52a-9577dba46928/payload-processing/0.log"